PowerDown to scale a build farm

Embedthis Software provides management software to device developers who create embedded management applications for their devices. Embedthis builds and tests their management software on an extensive build farm running an AWS CI/CD pipeline to ensure it is robust and secure.

But Embedthis needed to lower their EC2 build farm cloud costs without losing any flexibility and in-house solutions were only solving part of the problem.

Hosting the build farm in the cloud initially yielded big savings when switching from real hardware to virtual cloud servers, but the cost of running 60+ servers adds up. Embedthis needed to cut costs and PowerDown offered a solution to scale the farm using Spot instances and control it via dynamic scheduling to power down the build farm after-hours.

Cloud Budget Overflow

Embedthis runs up to 60 different build farm configurations on virtual machines running QEMU to emulate specific CPU architectures and operating systems. These configurations are used to build the device software and then run an exhaustive battery of regression and longevity tests. The build farm runs on a suite of AWS EC2 C4 instances behind an Auto Scale group which receives build requests that are routed to worker servers.

AWS charge for EC2 resources on a per-second basis. However, the Embedthis build farm is not required 24x7 and is typically not needed on weekends. While it is possible to manually turn on and off the farm each day, doing so consistently for a team has too much overhead. Embedthis, like many companies, has some remote staff who need access to the build farm at different and irregular times. A single fixed schedule would not suffice.

Scheduling Problems

PowerDown solved this problem for Embedthis via Dynamic Schedules where each developer nominates when they need the build farm and PowerDown determines the exact schedule. In this way, PowerDown’s dynamic schedules adapt and track the needs of the team. Remote users can enter their schedule using their local timezone. There is no need to call a meeting on slack or coordinate the schedule. PowerDown takes care of that.

Developers can easily override the schedule via the CLI or Web interface and extend the availability of the build farm if they need to work late.

The PowerDown CLI means a developer can simply type:

power up farm 4h

This will override the schedule and ensure the build farm is available for the next 4 hours.

Results

Embedthis software said: "With PowerDown, we are saving 65-75% of our cloud costs each month." "We tried manually turning off dev and test instances after hours, but this failed over time. PowerDown made this easy. It really is set and forget — PowerDown runs in the background and automatically turns off all our dev and test servers after hours and on weekends. It works well with our remote developers who are in a different timezone or when they have to work late with unexpected work hours."

Contact

Subscribe for News

This web site uses cookies to provide you with a better viewing experience. Without cookies, you will not be able to view videos, contact chat or use other site features. By continuing, you are giving your consent to cookies being used.