DistributedLog Operations ========================= Feature Provider ~~~~~~~~~~~~~~~~ DistributedLog uses a `feature-provider` library provided by Apache BookKeeper for managing features dynamically at runtime. It is a feature-flag_ system used to proportionally control what features are enabled for the system. In other words, it is a way of altering the control in a system without restarting it. It can be used during all stages of development, its most visible use case is on production. For instance, during a production release, you can enable or disable individual features, control the data flow through the system, thereby minimizing risk of system failure in real time. .. _feature-flag: https://en.wikipedia.org/wiki/Feature_toggle This `feature-provider` interface is pluggable and easy to integrate with any configuration management system. API ___ `FeatureProvider` is a provider that manages features under different scopes. The provider is responsible for loading features dynamically at runtime. A `Feature` is a numeric flag that control how much percentage of this feature will be available to the system - the number is called `availability`. :: Feature.name() => returns the name of this feature Feature.availability() => returns the availability of this feature Feature.isAvailable() => returns true if its availability is larger than 0; otherwise false It is easy to obtain a feature from the provider by just providing a feature name. :: FeatureProvider provider = ...; Feature feature = provider.getFeature("feature1"); // returns the feature named 'feature1' The `FeatureProvider` is scopable to allow creating features in a hierarchical way. For example, if a system is comprised of two subsystems, one is *cache*, while the other one is *storage*. so the features belong to different subsystems can be created under different scopes. :: FeatureProvider provider = ...; FeatureProvider cacheFeatureProvider = provider.scope("cache"); FeatureProvider storageFeatureProvider = provider.scope("storage"); Feature writeThroughFeature = cacheFeatureProvider.getFeature("write_through"); Feature duralWriteFeature = storageFeatureProvider.getFeature("dural_write"); // so the available features under `provider` are: (assume scopes are separated by '.') // - 'cache.write_through' // - 'storage.dural_write' The feature provider could be passed to `DistributedLogNamespaceBuilder` when building the namespace, thereby it would be used for controlling the features exposed under `DistributedLogNamespace`. :: FeatureProvider rootProvider = ...; FeatureProvider dlFeatureProvider = rootProvider.scope("dlog"); DistributedLogNamespace namespace = DistributedLogNamespaceBuilder.newBuilder() .uri(uri) .conf(conf) .featureProvider(dlFeatureProvider) .build(); The feature provider is loaded by reflection on distributedlog write proxy server. You could specify the feature provider class name as below. Otherwise it would use `DefaultFeatureProvider`, which disables all the features by default. :: featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider Configuration Based Feature Provider ____________________________________ Beside `DefaultFeatureProvider`, distributedlog also provides a file-based feature provider - it loads the features from properties files. All the features and their availabilities are configured in properties file format. For example, :: cache.write_through=100 storage.dural_write=0 You could configure `featureProviderClass` in distributedlog configuration file by setting it to `com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider` to enable file-based feature provider. The feature provider will load the features from two files, one is base config file configured by `fileFeatureProviderBaseConfigPath`, while the other one is overlay config file configured by `fileFeatureProviderOverlayConfigPath`. Current implementation doesn't differentiate these two files too much other than the `overlay` config will override the settings in `base` config. It is recommended to have a base config file for storing the default availability values for your system and dynamically adjust the availability values in overlay config file. :: featureProviderClass=com.twitter.distributedlog.feature.DynamicConfigurationFeatureProvider fileFeatureProviderBaseConfigPath=/path/to/base/config fileFeatureProviderOverlayConfigPath=/path/to/overlay/config // how frequent we reload the config files dynamicConfigReloadIntervalSec=60 Available Features __________________ Check the :doc:`../references/features` reference page for the features exposed by DistributedLog. `dlog` ~~~~~~ A CLI is provided for inspecting DistributedLog streams and metadata. .. code:: bash dlog JMX enabled by default Usage: dlog where command is one of: local Run distributedlog sandbox example Run distributedlog example tool Run distributedlog tool proxy_tool Run distributedlog proxy tool to interact with proxies balancer Run distributedlog balancer admin Run distributedlog admin tool help This help message or command is the full name of a class with a defined main() method. Environment variables: DLOG_LOG_CONF Log4j configuration file (default $HOME/src/distributedlog/distributedlog-service/conf/log4j.properties) DLOG_EXTRA_OPTS Extra options to be passed to the jvm DLOG_EXTRA_CLASSPATH Add extra paths to the dlog classpath These variable can also be set in conf/dlogenv.sh Create a stream _______________ To create a stream: .. code:: bash dlog tool create -u
-r -e List the streams ________________ To list all the streams under a given DistributedLog namespace: .. code:: bash dlog tool list -u
Show stream's information _________________________ To view the metadata associated with a stream: .. code:: bash dlog tool show -u
-s Dump a stream _____________ To dump the items inside a stream: .. code:: bash dlog tool dump -u
-s -o -l Delete a stream _______________ To delete a stream, run: .. code:: bash dlog tool delete -u
-s Truncate a stream _________________ Truncate the streams under a given DistributedLog namespace. You could specify a filter to match the streams that you want to truncate. There is a difference between the ``truncate`` and ``delete`` command. When you issue a ``truncate``, the data will be purge without removing the streams. A ``delete`` will delete the stream. You can pass the flag ``-delete`` to the ``truncate`` command to also delete the streams. .. code:: bash dlog tool truncate -u