Discover Kosmos and AWS

There is no denying that AWS is a market leader in the world of cloud computing — and only getting bigger. As part of our cloud journey at Discover Technologies, we have decided to ensure that one of our solutions, Discover Kosmos, could take advantage of the AWS platform without having to rearchitect the solution.

Discover Kosmos enables IT, compliance and records personnel to catalog and manage all of an organization’s digital assets from one platform. The Discover Kosmos dashboard provides a visual representation of all enterprise data on a single screen while allowing administrators to assess and act on data from the same interface.

The platform also enables a single application to search all repositories and conduct further business intelligence (BI) analysis. Discover Kosmos is built on a highly scalable microservices architecture that can be offered as a hosted service or on-premises deployment. We decided to package the solution using Docker containers leveraging Kubernetes to orchestrate the deployments. It is because of this that Discover Kosmos became the ideal candidate to be deployed on AWS.

The team at Discover Technologies leveraged another cloud provider as part of our DevOps pipeline. We knew the solution could already run both on premises and in the cloud using Docker and Kubernetes. As a matter of fact, our Development and QA instances were automatically deployed using a Continuous Integration pipeline.

During the second half of 2018, there was a desire to deploy Discover Kosmos on AWS. As mentioned earlier, this was determined to be technically feasible since we were utilizing Kubernetes for orchestration and could leverage a similar service (EKS) on AWS.

Next we needed to determine it was financially feasible (someone from Accounting is bound to come calling). The Amazon TCO Calculator was used to provide a simple comparison of physical on-premises deployment of the Discover Kosmos solution compared to a similar solution leveraging Amazon Web Services.

Further detail on expenditure was provided using the AWS Simply Monthly Calculator. Information garnered from these two resources, as well as previous experiences, was extrapolated to cover the entire life expectancy of the product to reinforce the cost benefit of AWS.

After the cost analysis was completed, it was time for the fun to begin and spin up Discover Kosmos on AWS. Below is a simple diagram of the deployment of Discover Kosmos on AWS.

The Discover Kosmos on AWS solution consists of a highly available Amazon Elastic Container Service for Kubernetes (EKS) cluster, deployed and exposed to the outside world using Elastic Load Balancing (ELB) and Route 53 domain mapping. Inside of the EKS cluster, ten worker nodes support a custom three node MongoDB cluster, as well as a three node ElasticSearch cluster with an equal number of master nodes.

Message traffic is managed via a Kafka cluster. All the persistence needs of the Discover Kosmos infrastructure and applications are handled by Amazon services, for example using Amazon Elastic Block Store (Amazon EBS) for Kubernetes persistent volumes. Finally, we are leveraging kubectl to deploy our application containers into the EKS cluster and expose it via ingress to the outside world.

One finding that we did encounter is that Amazon EKS does not auto-provision worker nodes. A bit of an inconvenience, but it was solved by provisioning them separately. Just a speed-bump, all else considered.

We were able to easily rehost Discover Kosmos on AWS without any loose of functionality or performance (if only it had solved some outstanding bugs for us, but we didn’t see Amazon promising development “miracles”). Currently our primary demonstration environment is hosted in AWS and we are looking forward to migrating additional workloads in the future.

The team is now looking into opportunities for a deeper AWS integration for Discover Kosmos. This includes leveraging AWS services to replace similar services currently in place for Discover Kosmos. The team believes that a deeper integration would improve performance and product reliability.

Thanks goes to the Discover Kosmos development team, with a special thank you to Steven Hooper for researching and implementing the deployment onto AWS.