Inside the most interesting challenges dev teams face — and the technology they use to overcome them

by April Bohnert

October 11, 2018

Both technology and the demands of tech consumers are constantly evolving, and with that comes an ever-changing landscape of opportunities and challenges for the companies on the front lines. For many engineering teams, the only way forward is through a combination of creative thinking, tech innovation and teamwork.

We asked four local tech leaders about the most interesting challenges their teams have faced recently, and the technology they used — or built from scratch — to overcome them.

Sovrn helps independent publishers and content creators grow their audiences, engage their readers and monetize their content with a suite of advertising and analytics tools. VP of Platform Engineering Kyle Gilliland shed some light on the technology his team used recently to overcome a big Big Data challenge.

What's an interesting technical challenge your team has faced recently?

Most recently, Sovrn engineered a transformation of our data ingest and processing capabilities. With over 4PB of data ingested per month and millisecond low-latency data requirements helping to run our ad-exchange, the challenge was to execute this without disruption of service and build a scalable solution that would meet our current and future business demands.

What technologies did you use to overcome this challenge?

Given the tight business timelines we have on data processing, data movement and analytics, we looked at solutions engaging micro-batching and elastic, cloud-based compute clusters.

One of the first steps was to evolve our ETL strategy into a formal data lake and separate our data storage from our data compute capabilities. We are now working to evaluate the way that we process data, leveraging technologies like Hive and Spark to find congruences within our data jobs and simplify the computing methods. By structuring our data in this manner, we have endless opportunities to mature and evolve the way data is provided to our business and customers.

Being in an industry of constant change means we need to create an environment that champions our engineers and staff to evolve and push the limits of technology.”

What process do you use for vetting and implementing new technologies? How do you help developers succeed when using technology they may not be familiar with?

When evaluating new technology, we start with the defined business needs, develop an understanding of available options, then work through various proofs of concept that derive a technology selection. Being in an industry of constant change means we need to create an environment that supports and champions our engineers and staff to evolve and push the limits of technology. At Sovrn, we cultivate a culture of learning to enable our engineers to explore, expand and grow their technical skills that bring value to our business.

Wunder Capital is driving the solar industry forward through its online investment and project management platform, which helps develop, manage and fund solar energy projects. Co-founder and CTO Dave Riess explained both the rewards and challenges of working with a largely proprietary software platform — and how his team continues to think outside the box to drive their product forward.

What's an interesting technical challenge your team has faced recently?

Our engineering team at Wunder has a fairly storied history of paving its own path on technical solutions, and our in-house accounting system is a great example. After evaluating the available off-the-shelf solutions, we decided to build our own double-entry accounting system from the ground up. This might seem pretty daunting, but one of our operating principles here at Wunder is “no problem is too hard.”

After pulling in some external resources to give our team a crash course in accounting, we were able to develop a design that excelled in all of the places where off-the-shelf solutions came up short. Our portfolio servicing infrastructure, which is supported by our custom accounting system, has become a huge asset to the business — in large part because our ability to iterate every aspect of the underlying software quickly enables our business team to make product choices that are not constrained by technical accounting limitations.

What technologies did you use to overcome this challenge?

Most of our systems are currently written in Ruby. After some fairly extensive experimentation, we’ve found an expressive, dynamic and mature environment like Ruby to be an incredibly powerful way for us to ship production code quickly and support rapid iteration.

We have found this to be especially true in contexts where some part of our problem domain is not yet well understood. Our codebase is technically a Rails application; however, as our system has matured, we have increasingly departed from Rails norms in ways that support maintainability and robustness in the context of a fairly large application. We often describe our application as “Rails but not Railsy.”

When vetting everything from engineering process iterations to technology choices to architectural decisions, we endeavor to be painfully explicit about what problem we are trying to solve...”

What process do you use for vetting and implementing new technologies? How do you help developers succeed when using technology they may not be familiar with?

When vetting everything from engineering process iterations to technology choices to architectural decisions, we endeavor to be painfully explicit about what problem we are trying to solve and then we focus exclusively on making a decision along dimensions that directly contribute to solving that problem.

When debating the relative merits of a set of options, it's easy to default to a complicated multivariate analysis along numerous dimensions of your various options, and in doing so, it's easy to lose track of the original question. It may sound pretty obvious, but making sure your problem statement is very well defined, and then keeping the conversation centered around that problem, is a very powerful way to make sound decisions.

Kapost’s B2B marketing platform enables businesses to map out and implement their content strategies, collaborate across teams and departments, and measure the success of their campaigns. Director of Data Science Elliot Cohen explained how his team has leveraged the latest technologies to automate and streamline previously manual user workflows.

What's an interesting technical challenge your team has faced recently?

Every day, marketers use Kapost to create, organize and track content. Part of the workflow is tagging each piece of new content with the initiatives, personas and strategies with which it is aligned.

Tagging is essential to successful content operations, but it can also be time-consuming. We are applying natural language processing and machine learning on previously tagged content to auto-suggest tags for new content. This has the potential to simultaneously streamline user workflows and improve alignment with initiatives.

What technologies did you use to overcome this challenge?

We are leveraging the speed and scalability of Spark to train hundreds of NLP models across our customer base. After training, we can serve predictions for new pieces of content with a lightweight Python Flask app.

We are leveraging the speed and scalability of Spark to train hundreds of [natural language processing] models across our customer base.”

What process do you use for vetting and implementing new technologies?

Before investing in a new technology, we implement a well-scoped, time-boxed engineering spike. The spike — or spikes — serve as a proof-of-concept and vetting mechanism for practical considerations such as software dependencies, usability and cost. If the spike is deemed successful, then we will proceed to a first-cut MVP.

Travelers Haven takes a tech-driven approach to short-term housing, combining the expertise and personal touch of a full-service housing firm with the convenience and speed of a tech platform. Director of Engineering John Siladie gave us insight into how his team is keeping its product on the cutting edge, even 10 years after its founding.

Tell us about an interesting technical challenge your dev team faced recently. What impact did that challenge have on the business?

We recently started working on a new initiative to help consumers with their short-term housing needs, and this new paradigm required us to think differently about our technology and product development workflow.

What technologies did you use to overcome this challenge?

We decided to write an entirely new front-end application in React for our consumer-focused product, which allowed us to iterate quickly on our MVP and get our application in front of our customers right away to get their feedback.

We learn together and share our findings to support each other when working with new technologies and frameworks...”

How do you help developers succeed when using technology they may not be familiar with?

Since React is a new technology for our team, we needed to ensure all our developers were on board quickly. One of our core values is “Never Failed, Only Learned.” We learn together and share our findings to support each other when working with new technologies and frameworks such as React.