Topics

Featured in Development

Understandability is the concept that a system should be presented so that an engineer can easily comprehend it. The more understandable a system is, the easier it will be for engineers to change it in a predictable and safe manner. A system is understandable if it meets the following criteria: complete, concise, clear, and organized.

Featured in Architecture & Design

Sonali Sharma and Shriya Arora describe how Netflix solved a complex join of two high-volume event streams using Flink. They also talk about managing out of order events and processing late arriving data, exploring keyed state for maintaining large state, fault tolerance of a stateful application, strategies for failure recovery, data validation batch vs streaming, and more.

Featured in Culture & Methods

Tim Cochran presents research gathered from ThoughtWorks' varied clients and projects, and shows some of the metrics their teams have identified as guides to creating the platform and the culture for high performing teams.

Feature flags are a great way to experiment with new features. For example, using feature flags you can roll out new features in a controlled way, or you can roll out different variations of a new feature to different user groups to better assess their experience. In addition, feature flags make it possible to quickly roll back a new feature that proved to be buggy or is not well accepted by your user base.

One important aspect of feature flag that is not often discussed, explains Uber engineer Murali Krishna Ramanathan, is technical debt, which can become quite large when feature flags are used extensively. The issue of technical debt arises when a feature flag comes to end of life, for example because a feature's gradual roll-out was completed, or because a feature was finally cancelled. When a feature flag becomes stale, all the code that implements it, as well all code that has become unreachable due to that flag removal, should be cleaned up:

Surprisingly, refactoring code and eliminating technical debt due to stale flags can be a non-trivial task. Our experience has shown that this is due to a confluence of diverse problems:

sub-optimal flag management that makes information on staleness of flag to be ambiguous,

churn in the organization affecting flag ownership, and

lack of developer incentives to cleanup source code related to the flags.

Uber engineers have come to manage over 6,000 flags across multiple apps and the time spent to clean up the code associated to obsolete flags could hinder developers from working on new flags, adds Ramanathan.

Piranha works at the abstract syntax tree (AST) level to detect a number of patterns that are usually associated to feature flags. Piranha is based on the use of specific annotations to identify feature flags in the code and on a flag management system to fully automate the process of stale code cleanup. To streamline the process for engineers, Piranha periodically queries the flag management system to know which flags have become stale, and then it generates a diff that is sent to the flag owner for review.

Uber found out that over 75% of all diffs generated by Piranha are effectively processed by engineers. Oddly, almost 95% of Objective-C diffs and 75% of Swift diffs are processed, while only round 55% of Java diffs are. This may be explained with the fact that Java diffs Piranha generates require more manual effort than either Swift or Objective-C diffs. Also, the more stringent app size requirements the App Store imposes on over-the-air updates adds to iOS engineers motivation to reduce their app binary sizes. Similarly, Piranha generates diffs that pass continuous integration testing with varying results depending on the language, with Objective-C diffs being almost always (99.41%) correct, and Java diffs the least (71.46%).

Piranha is not a perfect solution to the problem of stale flags and it has areas needing improvements, according to its creators. However, it proved to be effective among Uber's engineers in order to automate the process of paying out techical flag debt.