Here’s how we see it at Ladder — Historically, marketing has been predominantly driven by guesswork and luck based on what marketers think or have heard/read works best. But guesswork doesn’t lead to a repeatable, scalable model for growing a business.

Over the 2+ years since we started Ladder, we’ve been working to remove that guesswork from growth bit by bit.

A major next step for us in eradicating guesswork was to build proper machine learning directly into our proprietary technology to enable our team to make better strategic decisions driven by data.

We have a deep database of information about what marketing tactics have been successful, the ROI they’ve driven for clients, the industries and business models that they work on, and much, much more.

There’s so much that we know we can do with that data.

We have a talented product team with a functional knowledge of machine learning, so we built a basic version of machine learning into the Planner on our own to power a baseline recommendation engine for marketing tactics.

It was enough to prove out a minimum viable version, and after that it was time to bring on a team of domain experts.

To go above and beyond with a full machine learning implementation, we brought on the team at Data Revenue.

Follow along as we execute our 2017 Marketing Plan!
Sign up for the Ladder Weekly Playbook newsetter:

This is where we do all our work for clients. We plan monthly marketing sprints, enter all data on marketing tactics we run, and track and analyze the results of all the marketing tests our strategists execute, whether they’re successful, failed, or inconclusive.

The Planner is the technical backbone to our client strategy team, and as it gets better, so does our strategy, execution, and team.

Working with Data Revenue

Working with Data Revenue was an interesting experience for our product team. For this post I interviewed our co-founder, Michael Taylor, who leads the Product team, and our CTO, Adrian Slipko.

For Michael, who led the decision-making process on working with a data science / machine learning expert team, Data Revenue came across as very genuine compared to other machine learning vendors he spoke to.

The biggest positive sign was that they were straightforward in understanding our goal: Building a machine learning foundation now that will become superior in the long run, as we continue to gather more data from our work with clients.

Other vendors tried to sell us the dream and didn’t really come back with any hard information about what machine learning models they would use, while Data Revenue listened to our goals and provided model recommendations based on those goals.

Marketing Recommendation Engine

For Data Revenue, the task was straightforward:

Build a machine learning system using algorithms to recommend the right marketing tactics to strategists and companies, knowing right tactic for the right company at the right time. Beyond that, based on feedback, recommend even more tactics.

In interviewing Markus Schmitt, the founder of Data Revenue, I learned that what we had already was a strong baseline to work off of — we were already properly tracking a lot of pre-recommended tactics for a significant number of clients.

The question for his team was do we have enough data to take a machine learning algorithm and let it learn which tactics work with other tactics — basically creating a recommender system like Netflix (what you viewed, what you liked viewing, what else might you like?).

How we do strategy for any company isn’t by choosing a tactic at random. We look at the data and recommend strategy both based on understanding of company and on what tactics drive ROI. So for our machine learning implementation, every time intelligence is input into system, the recommender can use that to better automate recommendations, providing next-best tactics.

The result needed to be a smoother in-app interface for the Planner, providing recommendations that are more relevant and viable. The idea would be to make any job easier for any Ladder Strategist, new or old, to go through 1,000+ tactic database. No more going at a company with a blank slate, 15 auto-recommended based on company properties like industry, stage, etc…

Implementation

Implementing machine learning to create a full marketing recommendation engine for the Ladder Planner took Data Revenue 4 weeks. They went with an elegant approach based on the best-performing algorithm that could also be expanded on in the future. After testing a few different algorithms, they found the best approach would be an open-source model called LightFM.

According to Markus, evaluating a recommender system is not trivial. Data Revenue built custom evaluation functions that can tell you how good a recommender is without having to plug in a full dataset in order to test LightFM. The test was to see whether the algorithm was learning to rank the right recommendation types based on data inputs.

The final check was a subjective evaluation process that involved our co-founder Michael going in and doing a double-blind test on whether these recommendations would be ones he’d consider high-quality himself. Data Revenue gave him recommendations, he looked at the existing tactics used and the ones the algorithm recommended without knowing the name of the company, and asked himself “Would I have recommended these tactics based on the prior successful tactics?”

The accuracy of the algorithm was impressive: 95% of the time, Michael would have chosen the recommended tactic.

A second check then involved revealing the company name and looking at the recommendations again. Once again, 95% of tactics were ones he would have chosen.

Note: For Michael, the evaluation was whether each individual tactic was a good fit for the company. If he were to just come up with 10 tactics, they wouldn’t have been 95% the same. The algorithm is meant to answer the question “Are we recommending tactics that could work for this client?” From there, we can get more intricate with recommendations as the algorithm evolves.

Finally, everything was ready for us to fully implement the machine learning system into the Planner.

Upgrading the Ladder Planner

Our CTO, Adrian, worked directly with Data Revenue throughout the entire process, from creation to testing to implementation, in order to make sure we could easily implement their work into our platform.

His first step was to provide them with 100% anonymized data they could use to build the system. Data gathering was exported straight from our database, and he didn’t have to spend a long time preparing data for them. They just used what we had in our database, with no client and user names associated.

From that, they used the data we provided to pick out the best algorithm, and then had a discussion about how to implement it into the Planner. The result was an API that, when a company ID is given and a number of recommendations to display was set, the system returns recommended tactic IDs. We then handle displaying tactics on our side within the Planner.

The final product was built as a microservice that is hosted on Heroku and has an API that can be called, asking for next-best recommendation. The API is usable for anything on our side, from enriching recommendations in-app, to building minimal apps that surface top 20 recommendations for a strategist, and much more.

Limitations and Future Work

This is our first full implementation of machine learning into the Ladder Planner, so naturally we’re working continuously to expand and improve on it.

At the time of implementation, it was limited in that it wasn’t auto-updating, so our next major step was to build out an auto-updating system that would automatically make API calls when changes to things like company attributes, tactics executed, etc… occur.

As far as working with Data Revenue, contact with them directly was no issue, we had a good relationship, and there were no major issues or problems along the way. Response time from them was very good, usually taking up to 24 hours to respond. And best of all, they’re highly technical people that could talk to Adrian directly without needing to go through Michael, which was an invaluable time saver.

From here on, it’s all about evolving the machine learning system. The next step is “Are we recommending tactics that work better than what the strategist recommends.” We’re actively working on it, but it will require more data, hence why we’re focusing on growing our client base with industry-leading strategy.

For that, we have a strong relationship in place with experts that we’ve learned to trust.

If you’re looking to implement machine learning into your product, we’d highly recommend you use a service like Data Revenue to dip your toes in the water and get started today. You’ll learn a lot without taking on too many fixed costs – setting yourself up for future success.

Ladder USA

Ladder UK

Ladder Europe

Ladder is committed to protecting and respecting your privacy, and we’ll only use your personal information to administer your account and to provide services you requested from us. From time to time, we would like to contact you about relevant content that may be of interest to you. If you consent to us contacting you for this purpose, please tick below to say how you would like us to contact you:

I agree to receive relevant communications from Ladder. (required)

You may unsubscribe from these communications at any time. For more information on how to unsubscribe, our privacy practices, and how we are committed to protecting and respecting your privacy, please review our Privacy Policy.

By completing this form, you consent to allow Ladder to store and process the personal information submitted above to provide you the content and service requested.