A Day in the Life of Development - Server Management

What is your background and how long have you worked at Hopewiser?

Allan: My interest in computing started from an early age, when I wrote my first program at school. This continued through college and onto university where I completed a Computer Science degree. After university I took a software development position within the telecommunications sector and later joined Hopewiser as a senior software analyst. I have been at Hopewiser for 10+ years, working on a variety of projects from bespoke customer solutions to cloud based services.

Dimitrios: I joined Hopewiser in 2017. Prior to that, during my school years I worked part-time in Digitizing Documents for Government Projects. I have a Bachelor’s Degree in Software Engineering from Harokopio University in Athens, Greece, and a Master’s Degree in Network Systems / System Administration from the University of Sunderland.

Describe a typical day in your job?

Allan: A day’s work at Hopewiser can be quite varied. Normally, it would be spent developing our latest software products, where I could be involved in any part of the development life cycle from requirements capture through to implementation and release. Alternatively, it could be spent helping support with a customer query or investigating a server report.

Dimitrios: A typical day starts with checking reports and alerts from our internal and external monitoring tools. Scheduled tasks and scripts are all verified that they ran successfully during the night. If there is an error, we investigate to find the root cause and resolve as soon as possible. If everything goes smoothly, I devote most of my time into finding ways to improve the infrastructure, keep the systems secure and up to date, automate time-consuming processes, and making sure that our services and projects are compatible with the latest versions of runtime tools.

What do you enjoy most in your role?

Allan: The sense of achievement when a new product is rolled out and used by clients.

Dimitrios: Problem-solving. Administration of the systems requires a lot of problem-solving. It can be as simple as a scheduled task not running, a network connection dropping, or something more complex such as a server failure. You have to be alert, quick, and know where to look for answers. It is the most exciting feeling to see your solution taking effect and the system coming back to life.

What servers does Hopewiser use?

Allan: Hopewiser has a number of servers which each host dedicated applications. These are situated within the UK, EU, and Australia. The servers cover customer facing items, such as the websites and cloud based address capture APIs, plus network services like Firewalls, DNS, DHCP, and email. There are also servers hosting build machines that target different platforms and others hosting development and test environments.

How do you ensure a 99% availability for Hopewiser’s clients?

Allan: The system architecture used by Hopewiser’s cloud based services eliminate single point failures. The services are deployed on two independent servers that reside in different locations. These sit behind an externally hosted any-cast load balancer, acting as a worker/standby pair. Traffic is received by the client’s nearest load balancer and forwarded to the worker server. If the worker server fails then the load balancer will automatically direct traffic to the standby. If a load balancer fails then the next nearest one will receive the client traffic.

Dimitrios: Hopewiser employs a server failover to ensure maximum availability for our clients. To achieve that, we use Amazon Servers as a backup for Address Lookup services. When our failover mechanism detects an issue with the hosted servers, it automatically switches to the backup server in the Amazon Datacenter. Our systems are being centrally managed by Puppet, a configuration tool to describe what each end-point will do. On server failure, Puppet will allow us to recreate the system and have a fully-functioning server in under an hour. In the meantime, availability of the services are ensured through our Amazon Backup Servers.

How scaleable are the servers for Address Lookup?

Allan: If demand requires, additional servers could be added to the system with the load balancer configuration being changed to distribute traffic between them. To maintain integrity, each server should reside at independent locations and have sufficient capacity to cover the workload of a single failure. If an individual customer required high capacity then a duplicate infrastructure could be provided specifically for them.

Dimitrios: Scalability of our servers is always something we strive to do. As our user base increases, we as a team need to adapt into the needs of our customers. Last year, Hopewiser introduced their new Amazon Servers in Ireland and Sydney as an extension to our UK based Datacenter. This offers our UK and Australian costumers a dedicated server for their usage. Amazon allows the possibility to expand and contract to accommodate heavier or lighter loads, including, the ease with which a system or component can be modified, added, or removed, to accommodate changing loads. By introducing the Server Failover we have achieved the ability to enhance the system by adding new functionality without disrupting existing activities. Furthermore, the use of Puppet can provide us the ability to create new servers or add components to existing ones to increase functionality and achieve better load balancing.

Have you had to deal with any critical issues out of your control and if so, how quickly did you solve these?

Allan: Critical issues such as disk failures are something we need to be aware of. In the event of such a failure essential services are migrated to other servers whilst a replacement is sourced. Our servers are configured from a central location (master repository) which allows this to be done in a time efficient manner.

Dimitrios: There is one that comes to mind in particular, whereby we were able to rectify the issue in just shy of an hour.