Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

An Architecture for Agile Machine Learning in Real-Time Applications

Abstract of the paper:
Machine learning techniques have proved effective in recommender systems and other applications, yet teams working to deploy them lack many of the advantages that those in more established software disciplines today take for granted. The well-known Agile methodology advances projects in a chain of rapid development cycles, with subsequent steps often informed by production experiments. Support for such workflow in machine learning applications remains primitive.

The platform developed at if(we) embodies a specific machine learning approach and a rigorous data architecture constraint, so allowing teams to work in rapid iterative cycles. We require models to consume data from a time-ordered event history, and we focus on facilitating creative feature engineering. We make it practical for data scientists to use the same model code in development and in production deployment, and make it practical for them to collaborate on complex models.

We deliver real-time recommendations at scale, returning top results from among 10,000,000 candidates with sub-second response times and incorporating new updates in just a few seconds. Using the approach and architecture described here, our team can routinely go from ideas for new models to production-validated results within two weeks.

42.
• Single path to data for real-time streaming and history
• Shared feature engineering code for development and production
• Team shares access to code and data
• Fine-grained alignment of feature state and prediction outcomes
• Temporally accurate modeling ensured (no looking ahead)
Event History API

44.
• Open source implementation derived from
if(we)’s proprietary platform
• Provides Scala DSL for building online
features from event history
• Examples include dating recommendations
and product search with learning to rank
• Not yet ready for scale or production
• Seeking collaborators

46.
Agile Machine Learning with Event History
• Solving deployment yields quick product cycles
• All data saved and retrieved as time-ordered events
• Single path to data for both historical and real-time access
• Same feature engineering code used in development and production
• Agile success
• Team shares access to code and data
• Production product iterations measured in days rather than months
github.com/ifwe/antelope@jssmith