How EG Built a Modern, Event-Driven Architecture with MongoDB to Power Digital Transformation

UK’s Leading Commercial Property Data Service Delivers 50x More Releases Per Month, with 20x Improvement in Uptime

The total value of the UK commercial property is estimated at close to £1.7 trillion1. Investment decisions on big numbers requires big data, especially in handling a big variety of multi-structured data. And that is why EG, the UK’s leading commercial property data service, turned to MongoDB.

I met with Chris Fleetwood, Senior Director of Technology, and Chad Macey, Principal Architect at EG. We discussed how they are using agile methodologies with devops, cloud computing, and MongoDB as the foundation for the company’s digital transformation – moving from a century old magazine, into a data driven technology service.

Can you start by telling us a little bit about your company?

Building on over 150 years of experience, EG (formerly Estates Gazette) delivers market-leading data & analytics, insight, and decision support tools covering the UK commercial property market. Our services are used by real estate agents, lawyers, investors, surveyors, and property developers. We enable them to make faster, better informed decisions, and to win more business in the property market. We offer a comprehensive range of market data products with information on hundreds of thousands of properties across the UK, accessible across multiple channels including print, digital, online, and events. EG is part of Reed Business Information, providing data and analytics to business professionals around the world.

What is driving digital transformation in your business?

Our business was built on print media, with the Estates Gazette journal serving as the authoritative source on commercial property across the UK for well over a century. Back in the 1990s, we were quick to identify the disruptive potential of the Internet, embracing it as a new channel for information distribution. Pairing our rich catalog of property data and market intelligence with new data sources from mobile and location services – and the ability to run sophisticated analytics across all of it in the cloud – we are accelerating our move into enriched market insights, complemented with decision support systems.

IT was once just a supporting function for the traditional print media side of our business, but digital has now become our core engine for growth and revenue.

Can you describe your platform architecture?

Our data products are built on a microservices-inspired architecture that we call “pods”. Each pod serves a specific data product and audience. For example, the agent pod provides market intelligence for each geographic area such as recent sales, estimated values, local amenities, and zone regulations, along with lists of potential buyers and renters. Meanwhile the surveyor pod will maintain data used to generate valuations of true market worth. The pods also implement the business rules that drive the workflow for each of our different user audiences.

The advantage of our pod-based architecture is that it improves on our deployment and operational capabilities, supporting the transition to continuous delivery – giving us faster time to market for new features demanded by our customers. Each pod is owned by a “squad” with up to half a dozen members, comprising devops engineers, architects, and product managers.

Figure 1: EG Pod Architecture

How are you using MongoDB in this architecture?

Each pod maintains a local instance of the data it needs, pulling from a centralized system of record that stores all property details, transactions, locations, market data, availability, and so on. The system of record – or the “data-core pod” as we call it – in addition to each of the data product pods all run on MongoDB.

MongoDB is at the center of our event driven architecture. All updates are committed to the system of record – for example, when a new property comes onto the market – which then uses the Amazon Web Services (AWS) SNS push notification and SQS message queue services to publish the update to all the other product pods. This approach means all pods are working with the latest copy of the data, while avoiding tight coupling between each pod and the core database.

Why did you select MongoDB?

Agility and time to market are critical. We decided to use a Javascript-based technology stack that allows consistent developer experience from the client, to the server, through to the database, without having to deal with any translations between layers.

We evaluated multiple database options as part of our platform modernization:

Having used relational databases extensively in the past, we knew the pain and friction involved with having to define a schema in the database, and then re-implement that same schema again in an ORM at the application layer. And we would need to repeat this process for each pod we developed, and for each change in the data model as we evolved application functionality.

We also took a look at an alternative NoSQL document database, but it did not provide the development speed we needed as we found it was far too complex and difficult to use.

As the M in the MEAN stack, we knew MongoDB would work well with Javascript and Node.js. I spun up a local instance on my laptop, and was up and running in less that 5 minutes, and productive within the hour. I judge all technology by my one hour rule. Basically, if within an hour I can start to understand and be productive with the technology, that tells me it's got a really good developer experience, supported by comprehensive documentation. If it's harder than that, I’m not likely to get along with that technology in the longer term. We didn’t look back from that point onwards – we put MongoDB through its paces to ensure it delivered the schema flexibility, query functionality, performance, and scale we needed, and it passed all of our tests.

Can you describe your MongoDB deployment?

We run MongoDB Enterprise Advanced on AWS. We get access to MongoDB support, in addition to advanced tooling. We are in the process of installing Ops Manager to take advantage of fine-grained monitoring telemetry delivered to our devops team. This insight enables them to continue to scale the service as we onboard more data products and customers.

MongoDB Compass is a new tool we’ve started evaluating. The schema visualizations can help our developers to explore and optimize each pod’s data model. The new integrated geospatial querying capability is especially valuable for our research teams. They can use the Compass GUI to construct sophisticated queries with a simple point and click interface, returning results graphically, and as sets of JSON documents. Without this functionality, our valuable developer resource would have been tied up creating the queries for them.

We will also be upgrading to the latest MongoDB release to take advantage of the MongoDB Encrypted storage engine to extend our security profile, and prepare for the new EU General Data Protection Regulation (GDPR) market legislation that is coming into force in 2018.

How is MongoDB performing for you?

MongoDB has been rock solid for us. With very few operational issues our team is able to focus on building new products. The flexible data model, rich query language, and indexing makes development super-fast. Geospatial search is the starting point for the user experience – a map is the first thing a customer sees when they access our data products. MongoDB’s geospatial queries and indexes allow our customers to easily navigate market data by issuing polygon searches that display properties within specific coordinates of interest.

We also depend on the MongoDB aggregation pipeline to generate the analytics and insights the business, and our customers, rely on. For example, we can quickly generate averages for rents achieved in a specific area, aggregated against the total number of transactions over a given time period. Each MongoDB release enriches the aggregation pipeline, so we always have new classes of analytics we can build on top of the database.

How are you measuring the impact of MongoDB on your business?

It’s been a core part of our team transitioning from being a business support function to being an enabler of new revenue streams. With our pod-based architecture powered by MongoDB, we can get products to market faster, and release more frequently.

With relational databases, we were constrained to one new application deployment per month. With MongoDB, we are deploying 50 to 60 times per month. With MongoDB’s self-healing replica set architecture, we’ve delivered 20x improvement in uptime compared to the previous generation of systems.

Chris and Chad, thanks for taking the time to share your experiences with the MongoDB community