0.8.2

Added backtype.storm.scheduler.IsolationScheduler. This lets you run topologies that are completely isolated at the machine level. Configure Nimbus to isolate certain topologies, and how many machines to give to each of those topologies, with the isolation.scheduler.machines config in Nimbus's storm.yaml. Topologies run on the cluster that are not listed there will share whatever remaining machines there are on the cluster after machines are allocated to the listed topologies.

Storm UI now uses nimbus.host to find Nimbus rather than always using localhost (thanks Frostman)

Added report-error! to Clojure DSL

Automatically throttle errors sent to Zookeeper/Storm UI when too many are reported in a time interval (all errors are still logged) Configured with TOPOLOGY_MAX_ERROR_REPORT_PER_INTERVAL and TOPOLOGY_ERROR_THROTTLE_INTERVAL_SECS

Kryo instance used for serialization can now be controlled via IKryoFactory interface and TOPOLOGY_KRYO_FACTORY config

Add ability to plug in custom code into Nimbus to allow/disallow topologies to be submitted via NIMBUS_TOPOLOGY_VALIDATOR config

Added TOPOLOGY_TRIDENT_BATCH_EMIT_INTERVAL_MILLIS config to control how often a batch can be emitted in a Trident topology. Defaults to 500 milliseconds. This is used to prevent too much load from being placed on Zookeeper in the case that batches are being processed super quickly.

Log any topology submissions errors in nimbus.log

Add static helpers in Config when using regular maps

Make Trident much more memory efficient during failures by immediately removing state for failed attempts when a more recent attempt is seen

Add ability to name portions of a Trident computation and have those names appear in the Storm UI

Show Nimbus and topology configurations through Storm UI (thanks rnfein)

Added ITupleCollection interface for TridentState's and TupleCollectionGet QueryFunction for getting the full contents of a state. MemoryMapState and LRUMemoryMapState implement this

Can now submit a topology in inactive state. Storm will wait to call open/prepare on the spouts/bolts until it is first activated.

Can now activate, deactive, rebalance, and kill topologies from the Storm UI (thanks Frostman)

Can now use --config option to override which yaml file from ~/.storm to use for the config (thanks tjun)

Major optimization for unreliable spouts and unanchored tuples (will use far less CPU)

Revamped internals of Storm to use LMAX disruptor for internal queuing. Dramatic reductions in contention and CPU usage.

Numerous micro-optimizations all throughout the codebase to reduce CPU usage.

Optimized internals of Storm to use much fewer threads - two fewer threads per spout and one fewer thread per acker.

Removed error method from task hooks (to be re-added at a later time)

Validate that subscriptions come from valid components and streams, and if it's a field grouping that the schema is correct (thanks xumingming)

MemoryTransactionalSpout now works in cluster mode

Only track errors on a component by component basis to reduce the amount stored in zookeeper (to speed up UI). A side effect of this change is the removal of the task page in the UI.

Add TOPOLOGY-TICK-TUPLE-FREQ-SECS config to have Storm automatically send “tick” tuples to a bolt's execute method coming from the __system component and __tick stream at the configured frequency. Meant to be used as a component-specific configuration.

Upgrade Kryo to v2.17

Tuple is now an interface and is much cleaner. The Clojure DSL helpers have been moved to TupleImpl

Added shared worker resources. Storm provides a shared ExecutorService thread pool by default. The number of threads in the pool can be configured with topology.worker.shared.thread.pool.size

Improve CustomStreamGrouping interface to make it more flexible by providing more information

0.7.3

Changed debug level of “Failed message” logging to DEBUG

Bug fix: Fixed critical regression in 0.7.2 that could cause workers to timeout to the supervisors or to Nimbus. 0.7.2 moved all system tasks to the same thread, so if one took a long time it would block the other critical tasks. Now different system tasks run on different threads.

0.7.2

NOTE: The change from 0.7.0 in which OutputCollector no longer assumes immutable inputs has been reverted to support optimized sending of tuples to colocated tasks

Messages sent to colocated tasks are sent in-memory, skipping serialization (useful in conjunction with localOrShuffle grouping) (thanks xumingming)

Bug fix: Fixed race condition in FeederSpout that could lead to dropped messages

Bug fix: Quoted arguments with spaces now work properly with storm client script

Bug fix: Workers start properly when topology name has spaces

Bug fix: UI works properly when there are spaces in topology or spout/bolt names (thanks xiaokang)

Bug fix: Tuple$Seq now returns correct count (thanks travisfw)

0.7.0

Transactional topologies: a new higher level abstraction that enables exactly-once messaging semantics for most computations. Documented on the wiki.

Component-specific configurations: Can now set configurations on a per-spout or per-bolt basis.

New batch bolt abstraction that simplifies the processing of batches in DRPC or transactional topologies. A new batch bolt is created per batch and they are automatically cleaned up.

Introduction of base classes for various bolt and spout types. These base classes are in the backtype.storm.topology.base package and provide empty implementations for commonly unused methods

CoordinatedBolt generalized to handle non-linear topologies. This will make it easy to implement a non-linear DRPC topology abstraction.

Can customize the JVM options for Storm UI with new ui.childopts config

BigIntegers are now serializable by default

All bolts/spouts now emit a system stream (id “__system”). Currently it only emits startup events, but may emit other events in the future.

Optimized tuple trees for batch processing in DRPC and transactional topologies. Only the coordination tuples are anchored. OutputCollector#fail still works because CoordinatedBolt will propagate the fail to all other tuples in the batch.

CoordinatedBolt moved to backtype.storm.coordination package

Clojure test framework significantly more composable

Massive internal refactorings and simplifications, including changes to the Thrift definition for storm topologies.

Optimized acking system. Bolts with zero or more than one consumer used to send an additional ack message. Now those are no longer sent.

Changed interface of CustomStreamGrouping to receive a List rather than a Tuple.

Bug fix: Multilang resources are now extracted from the relevant jar on the classpath when appropriate. Previously an error would be thrown if the resources/ dir was in a jar in local mode.

Bug fix: Fix race condition in unit testing where time simulation fails to detect that Storm cluster is waiting due to threads that are not alive

Bug fix: Fix deadlock in Nimbus that could be triggered by a kill command.

0.6.1

storm client “activate” and “deactivate” commands

storm client “rebalance” command

Nimbus will automatically detect and cleanup corrupt topologies (this would previously give an error of the form “file storm...ser cannot be found”).

“storm” client will not run unless it's being used from a release.

Topology jar path now passed in using a java property rather than an environment variable.

LD_LIBRARY_PATH environment variable is now set on worker processes appropriately.

Replaced jvyaml with snakeyaml. UTF-8 YAML files should now work properly.

Upgraded httpclient, httpcore, and commons-codec dependencies.

0.6.0

New serialization system based on Kryo

Component and stream ids are now strings

Pluggable stream groupings

Storm now chooses an unused port for Zookeeper in local mode instead of crashing when 2181 was in use.

Better support for defining topologies in non-JVM languages. The Thrift structure for topologies now allows you to specify components using a Java class name and a list of arguments to that class's constructor.

Bug fix: errors during the preparation phase of spouts or bolts will be reported to the Storm UI