Tuesday, December 22, 2015

Improvements to Uptime in Progress

Patchrends.com is the World's largest price database for Boy Scout patches, Order of the Arrow flaps, and many other pieces of Scouting's collectibles.

Naturally, collecting thousands of price points everyday directly from eBay presents a number of interesting challenges. So does storing millions of database records and images. But, that's why I remain motivated to devote the time and energy into this invaluable service.

The Patchtrends.com Infrastructure

The service is divided up into several different pieces. The only piece that members and visitors get to see is the website. The website is hosted on a computer that is usually referred to as a "web server." The other computers that support Patchtrends.com are focused on collecting data, organizing this data, and presenting a stable and reliable storage platform for the millions of data points and item images.

At the time of this writing, there is almost 1 TB (that's one thousand GB) of data collected. The vast majority of this data is made up of item images. And this is growing every day. Due to a lot of hard work, the site has room to grow into several terabytes of data before I have to consider expanding. Thankfully, storage mediums are cheap and open source software tools exist for making this backend scale nearly indefinitely. If there is interest in me expanding more on the technical aspects of the site and the infrastructure supporting it, I'd be more than happy to oblige.

Embracing the Cloud

As many of you may know, the webserver hosting Patchtrends.com has been at the mercy of flaky power at its colocation facility for some time now. I've been resistant in the past to pay additional fees for the
hosting of the website itself, but I finally had enough of the stability
issues the webserver was facing.

Recently, I took a major step in mitigating this major annoyance by migrating Patchtrends.com to a Digital Ocean, a provider of affordable virtual infrastructure. I looked at plans also for Amazon's EC2/AWS services, and determined Digital Ocean was a better fit for n ow.

While this move has not been without its technical challenges, it's almost mostly completed. Much of the work was done to ensure that the webserver, once physically connected to the data collection and storage infrastructure, would be able to have fast access to the data no matter where the "server" was located.

This has been solved for the textual pricing data, but not yet for the millions of images that can't possibly be stored in the "cloud." A solution is on the board, but it needs to be implemented. It's not all that involved, but will take some time to get right.

Goal: Zero Downtime

Zero downtime is not an unattainable goal for Patchtrends.com. In fact, it's very possible; and we've started down the path to achieving this state.

The end result of the move to the cloud is that the site is now much easier to replicate on a remote webserver. What is means rather is that I can build out multiple backup webservers that are able to mask the unlikely situation where any one of the servers becomes unavailable.

I am currently working on setting up a backup server through the services provided by Scaleway.com. Whereas Digital Ocean is located in New York City (at least my instance is), Scaleway houses its infrastructure in Paris. If the server is NY goes down for some reason (e.g., another Hurricane Sandy), then visitors will be directed to the Paris server without even knowing. Having more servers just makes our resiliency even greater.

And the additional servers don't just mean failover from unforeseen outages. The same cluster of replicated servers can be used to balance out the load being put onto it from high periods of usage - a nice problem to have!

Patchtrends Needs Your Support

Expanding our infrastructure by using paid services is a new expense for Patchtrends.com. In order to help keep prices ridiculously low, please support my efforts by visiting the following eBayaffiliate links presented for active items for sale at Patchbazaar.com.

In Conclusion

Patchtrends.com is poised to be ushered into modern capabilities - resiliency, high availability, and scalability. We've just taken a small, but critical step in this direction. Stay tuned for more!

No comments:

Post a Comment

Follow by Email

About Me

I have been collecting patches since I was bitten by the bug at a Conclave in 1991. I am an Eagle Scout, Vigil Honor inductee and former Lodge Chief (#397), and have spent countless nights under the stars.

My passion also extends to computers and programming. As such, I set out in 2013 to create something truly useful.

By utilizing my background in programming and system administration, I created Patchtrends.com with the specific goal of providing the Scout Patch Collecting community the very best, largest price database and research platform possible.

My overall guiding principles can be summed up as:

Obey the Scout Oath and Scout Law

Provide an exception, cutting edge product

Provide exception customer service and focus on customer satisfaction

Promote both patch trading and technology to our Youth

Leave this hobby in a better place than when I found it

I am always looking for new and innovative ways to apply the latest technologies and development in "big data" to the ever growing massive database of Scouting Memorabilia sales data.