DistributedLog

git@github.com:twitter/distributedlog.git/tree/master/
OWNERS None
TAGS #uses_maven
HEALTH

9.0 / 10

RATING

Cluster Setup & Deployment

This section describes how to run DistributedLog in distributed mode. To run a cluster with DistributedLog, you need a Zookeeper cluster and a Bookkeeper cluster.

Build

To build DistributedLog, run:

mvn clean install -DskipTests

Or run ./scripts/snapshot to build the release packages from current source. The released packages contain the binaries for running distributedlog-service, distributedlog-benchmark and distributedlog-tutorials.

NOTE: we run following instructions from distributedlog source code after running mvn clean install. And assume DLOG_HOME is the directory of distributedlog source.

Zookeeper

(If you already have a zookeeper cluster running, you could skip this section.)

We could use the dlog-daemon.sh and the zookeeper.conf.template to demonstrate run a 1-node zookeeper ensemble locally.

Create a zookeeper.conf from the zookeeper.conf.template.

$ cp distributedlog-service/conf/zookeeper.conf.template distributedlog-service/conf/zookeeper.conf

Configure the settings in zookeeper.conf. By default, it will use /tmp/data/zookeeper for storing the zookeeper data. Let's create the data directories for zookeeper.

$ mkdir -p /tmp/data/zookeeper/txlog

Once the data directory is created, we need to assign myid for this zookeeper node.

$ echo "1" > /tmp/data/zookeeper/myid

Start the zookeeper daemon using dlog-daemon.sh.

$ ./distributedlog-service/bin/dlog-daemon.sh start zookeeper ${DL_HOME}/distributedlog-service/conf/zookeeper.conf

You could verify the zookeeper setup using zkshell.

// ./distributedlog-service/bin/dlog zkshell ${zkservers}
$ ./distributedlog-service/bin/dlog zkshell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]
[zk: localhost:2181(CONNECTED) 1]

Please refer to the ZooKeeper for more details on setting up zookeeper cluster.

Bookkeeper

(If you already have a bookkeeper cluster running, you could skip this section.)

We could use the dlog-daemon.sh and the bookie.conf.template to demonstrate run a 3-nodes bookkeeper cluster locally.

Create a bookie.conf from the bookie.conf.template. Since we are going to run a 3-nodes bookkeeper cluster locally. Let's make three copies of bookie.conf.template.

$ cp distributedlog-service/conf/bookie.conf.template distributedlog-service/conf/bookie-1.conf
$ cp distributedlog-service/conf/bookie.conf.template distributedlog-service/conf/bookie-2.conf
$ cp distributedlog-service/conf/bookie.conf.template distributedlog-service/conf/bookie-3.conf

Configure the settings in the bookie configuraiont files.

First of all, choose the zookeeper cluster that the bookies will use and set zkServers in the configuration files.

zkServers=localhost:2181

Choose the zookeeper path to store bookkeeper metadata and set zkLedgersRootPath in the configuration files. Let's use /messaging/bookkeeper/ledgers in this instruction.

zkLedgersRootPath=/messaging/bookkeeper/ledgers

Format bookkeeper metadata

(NOTE: only format bookkeeper metadata when first time setting up the bookkeeper cluster.)

The bookkeeper shell doesn't automatically create the zkLedgersRootPath when running metaformat. So using zkshell to create the zkLedgersRootPath.

$ ./distributedlog-service/bin/dlog zkshell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] create /messaging ''
Created /messaging
[zk: localhost:2181(CONNECTED) 1] create /messaging/bookkeeper ''
Created /messaging/bookkeeper
[zk: localhost:2181(CONNECTED) 2] create /messaging/bookkeeper/ledgers ''
Created /messaging/bookkeeper/ledgers
[zk: localhost:2181(CONNECTED) 3]

If the zkLedgersRootPath, run metaformat to format the bookkeeper metadata.

$ BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-1.conf ./distributedlog-service/bin/dlog bkshell metaformat
Are you sure to format bookkeeper metadata ? (Y or N) Y

Add Bookies

Once the bookkeeper metadata is formatted, it is ready to add bookie nodes to the cluster.

Configure Ports

Configure the ports that used by bookies.

bookie-1:

# Port that bookie server listen on
bookiePort=3181
# Exporting codahale stats
185 codahaleStatsHttpPort=9001

bookie-2:

# Port that bookie server listen on
bookiePort=3182
# Exporting codahale stats
185 codahaleStatsHttpPort=9002

bookie-3:

# Port that bookie server listen on
bookiePort=3183
# Exporting codahale stats
185 codahaleStatsHttpPort=9003

Configure Disk Layout

Configure the disk directories used by a bookie server by setting following options.

# Directory Bookkeeper outputs its write ahead log
journalDirectory=/tmp/data/bk/journal
# Directory Bookkeeper outputs ledger snapshots
ledgerDirectories=/tmp/data/bk/ledgers
# Directory in which index files will be stored.
indexDirectories=/tmp/data/bk/ledgers

As we are configuring a 3-nodes bookkeeper cluster, we modify the following settings as below:

bookie-1:

# Directory Bookkeeper outputs its write ahead log
journalDirectory=/tmp/data/bk-1/journal
# Directory Bookkeeper outputs ledger snapshots
ledgerDirectories=/tmp/data/bk-1/ledgers
# Directory in which index files will be stored.
indexDirectories=/tmp/data/bk-1/ledgers

bookie-2:

# Directory Bookkeeper outputs its write ahead log
journalDirectory=/tmp/data/bk-2/journal
# Directory Bookkeeper outputs ledger snapshots
ledgerDirectories=/tmp/data/bk-2/ledgers
# Directory in which index files will be stored.
indexDirectories=/tmp/data/bk-2/ledgers

bookie-3:

# Directory Bookkeeper outputs its write ahead log
journalDirectory=/tmp/data/bk-3/journal
# Directory Bookkeeper outputs ledger snapshots
ledgerDirectories=/tmp/data/bk-3/ledgers
# Directory in which index files will be stored.
indexDirectories=/tmp/data/bk-3/ledgers

Format bookie

Once the disk directories are configured correctly in the configuration file, use bkshell bookieformat to format the bookie.

BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-1.conf ./distributedlog-service/bin/dlog bkshell bookieformat
BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-2.conf ./distributedlog-service/bin/dlog bkshell bookieformat
BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-3.conf ./distributedlog-service/bin/dlog bkshell bookieformat

Start bookie

Start the bookie using dlog-daemon.sh.

SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-1.conf
SERVICE_PORT=3182 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-2.conf
SERVICE_PORT=3183 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-3.conf

Verify whether the bookie is setup correctly. You could simply check whether the bookie is showed up in zookeeper zkLedgersRootPath/available znode.

$ ./distributedlog-service/bin/dlog zkshell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /messaging/bookkeeper/ledgers/available
[127.0.0.1:3181, 127.0.0.1:3182, 127.0.0.1:3183, readonly]
[zk: localhost:2181(CONNECTED) 1]

Or check if the bookie is exposing the stats at port codahaleStatsHttpPort.

// ping the service
$ curl localhost:9001/ping
pong
// checking the stats
curl localhost:9001/metrics?pretty=true

Stop bookie

Stop the bookie using dlog-daemon.sh.

$ ./distributedlog-service/bin/dlog-daemon.sh stop bookie
// Example:
$ SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh stop bookie
doing stop bookie ...
stopping bookie
Shutdown is in progress... Please wait...
Shutdown completed.

Turn bookie to readonly

Start the bookie in readonly mode.

$ SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-1.conf --readonly

Verify if the bookie is running in readonly mode.

$ ./distributedlog-service/bin/dlog zkshell localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /messaging/bookkeeper/ledgers/available
[127.0.0.1:3182, 127.0.0.1:3183, readonly]
[zk: localhost:2181(CONNECTED) 1] ls /messaging/bookkeeper/ledgers/available/readonly
[127.0.0.1:3181]
[zk: localhost:2181(CONNECTED) 2]

Please refer to the BookKeeper for more details on setting up bookkeeper cluster.

Create Namespace

After setting up a zookeeper cluster and a bookkeeper cluster, you could provision DL namespaces for applications to use.

Provisioning a DistributedLog namespace is accomplished via the bind command available in dlog tool.

Namespace is bound by writing bookkeeper environment settings (e.g. the ledger path, bkLedgersZkPath, or the set of Zookeeper servers used by bookkeeper, bkZkServers) as metadata in the zookeeper path of the namespace DL URI. The DL library resolves the DL URI to determine which bookkeeper cluster it should read and write to.

The namespace binding has following features:

  • Inheritance: suppose distributedlog://<zkservers>/messaging/distributedlog is bound to bookkeeper cluster X. All the streams created under distributedlog://<zkservers>/messaging/distributedlog, will write to bookkeeper cluster X.
  • Override: suppose distributedlog://<zkservers>/messaging/distributedlog is bound to bookkeeper cluster X. You want streams under distributedlog://<zkservers>/messaging/distributedlog/S write to bookkeeper cluster Y. You could just bind distributedlog://<zkservers>/messaging/distributedlog/S to bookkeeper cluster Y. The binding to distributedlog://<zkservers>/messaging/distributedlog/S only affects streams under distributedlog://<zkservers>/messaging/distributedlog/S.

Create namespace binding using dlog tool. For example, we create a namespace distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace pointing to the bookkeeper cluster we just created above.

$ distributedlog-service/bin/dlog admin bind \\
    -dlzr 127.0.0.1:2181 \\
    -dlzw 127.0.0.1:2181 \\
    -s 127.0.0.1:2181 \\
    -bkzr 127.0.0.1:2181 \\
    -l /messaging/bookkeeper/ledgers \\
    -i false \\
    -r true \\
    -c \\
    distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace

No bookkeeper is bound to distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace
Created binding on distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace.
  • Configure the zookeeper cluster used for storing DistributedLog metadata: -dlzr and -dlzw. Ideally -dlzr and -dlzw would be same the zookeeper server in distributedlog namespace uri. However to scale zookeeper reads, the zookeeper observers sometimes are added in a different domain name than participants. In such case, configuring -dlzr and -dlzw to different zookeeper domain names would help isolating zookeeper write and read traffic.
  • Configure the zookeeper cluster used by bookkeeper for storing the metadata : -bkzr and -s. Similar as -dlzr and -dlzw, you could configure the namespace to use different zookeeper domain names for readers and writers to access bookkeeper metadatadata.
  • Configure the bookkeeper ledgers path: -l.
  • Configure the zookeeper path to store DistributedLog metadata. It is implicitly included as part of namespace URI.

Write Proxy

A write proxy consists of multiple write proxies. They don't store any state locally. So they are mostly stateless and can be run as many as you can.

Configuration

Different from bookkeeper, DistributedLog tries not to configure any environment related settings in configuration files. Any environment related settings are stored and configured via namespace binding. The configuration file should contain non-environment related settings.

There is a write_proxy.conf template file available under distributedlog-service module.

Run write proxy

A write proxy could be started using dlog-daemon.sh script under distributedlog-service.

WP_SHARD_ID=${WP_SHARD_ID} WP_SERVICE_PORT=${WP_SERVICE_PORT} WP_STATS_PORT=${WP_STATS_PORT} ./distributedlog-service/bin/dlog-daemon.sh start writeproxy
  • WP_SHARD_ID: A non-negative integer. You don't need to guarantee uniqueness of shard id, as it is just an indicator to the client for routing the requests. If you are running the write proxy using a cluster scheduler like aurora, you could easily obtain a shard id and use that to configure WP_SHARD_ID.
  • WP_SERVICE_PORT: The port that write proxy listens on.
  • WP_STATS_PORT: The port that write proxy exposes stats to a http endpoint.

Please check distributedlog-service/conf/dlogenv.sh for more environment variables on configuring write proxy.

  • WP_CONF_FILE: The path to the write proxy configuration file.
  • WP_NAMESPACE: The distributedlog namespace that the write proxy is serving for.

For example, we start 3 write proxies locally and point to the namespace created above.

$ WP_SHARD_ID=1 WP_SERVICE_PORT=4181 WP_STATS_PORT=20001 ./distributedlog-service/bin/dlog-daemon.sh start writeproxy
$ WP_SHARD_ID=2 WP_SERVICE_PORT=4182 WP_STATS_PORT=20002 ./distributedlog-service/bin/dlog-daemon.sh start writeproxy
$ WP_SHARD_ID=3 WP_SERVICE_PORT=4183 WP_STATS_PORT=20003 ./distributedlog-service/bin/dlog-daemon.sh start writeproxy

The write proxy will announce itself to the zookeeper path .write_proxy under the dl namespace path.

We could verify that the write proxy is running correctly by checking the zookeeper path or checking its stats port.

::

$ ./distributedlog-service/bin/dlog zkshell localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled

WATCHER:

WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] ls /messaging/distributedlog/mynamespace/.write_proxy [member_0000000000, member_0000000001, member_0000000002]

$ curl localhost:20001/ping
pong

Add and Remove Write Proxies

Removing a write proxy is pretty straightforward by just killing the process.

WP_SHARD_ID=1 WP_SERVICE_PORT=4181 WP_STATS_PORT=10001 ./distributedlog-service/bin/dlog-daemon.sh stop writeproxy

Adding a new write proxy is just adding a new host and starting the write proxy process as described above.

Write Proxy Naming

The dlog-daemon.sh script starts the write proxy by announcing it to the .write_proxy path under the dl namespace. So you could use zk!<zkservers>!/<namespace_path>/.write_proxy as the finagle name to access the write proxy cluster. It is zk!127.0.0.1:2181!/messaging/distributedlog/mynamespace/.write_proxy in the above example.

Verify the setup

You could verify the write proxy cluster by running tutorials over the setup cluster.

Create 10 streams.

$ ./distributedlog-service/bin/dlog tool create -u distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace -r stream- -e 0-10
You are going to create streams : [stream-0, stream-1, stream-2, stream-3, stream-4, stream-5, stream-6, stream-7, stream-8, stream-9, stream-10] (Y or N) Y

Tail read from the 10 streams.

$ ./distributedlog-tutorials/distributedlog-basic/bin/runner run c.twitter.distributedlog.basic.MultiReader distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace stream-0,stream-1,stream-2,stream-3,stream-4,stream-5,stream-6,stream-7,stream-8,stream-9,stream-10

Run record generator over some streams

$ ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.RecordGenerator 'zk!127.0.0.1:2181!/messaging/distributedlog/mynamespace/.write_proxy' stream-0 100
$ ./distributedlog-tutorials/distributedlog-basic/bin/runner run com.twitter.distributedlog.basic.RecordGenerator 'zk!127.0.0.1:2181!/messaging/distributedlog/mynamespace/.write_proxy' stream-1 100

Check the terminal running MultiReader. You will see similar output as below:

"""
Received record DLSN{logSegmentSequenceNo=1, entryId=21044, slotId=0} from stream stream-0
"""
record-1464085079105
"""
Received record DLSN{logSegmentSequenceNo=1, entryId=21046, slotId=0} from stream stream-0
"""
record-1464085079113
"""
Received record DLSN{logSegmentSequenceNo=1, entryId=9636, slotId=0} from stream stream-1
"""
record-1464085079110
"""
Received record DLSN{logSegmentSequenceNo=1, entryId=21048, slotId=0} from stream stream-0
"""
record-1464085079125
"""
Received record DLSN{logSegmentSequenceNo=1, entryId=9638, slotId=0} from stream stream-1
"""
record-1464085079121
"""
Received record DLSN{logSegmentSequenceNo=1, entryId=21050, slotId=0} from stream stream-0
"""
record-1464085079133
"""
Received record DLSN{logSegmentSequenceNo=1, entryId=9640, slotId=0} from stream stream-1
"""
record-1464085079130
"""

Please refer to the Performance Tuning for more details on tuning performance.