Thursday 15th September 2011

Maybe you were one of the lucky customers to receive Mint.com's 6am "oops" emails from staging servers in October 2010, or perhaps you work with data regulated by HIPAA - either way, you know it's not cool to have copies of production data anywhere that's not production. We'll cover strategies to protect and appropriately slice your data for use in testing and development environments.

Friday 16th September 2011

Wal-E is an open source tool designed to continuously archive your business's Postgres data to Amazon's S3 cloud storage service. It can help with recovery from failures even when the individual database servers are not individually very reliable, which is the case in many common cloud database deployments.

Individual systems in any environment can't be relied on to save critical data perfectly. No matter how much reundancy your database server includes, you still need to save backups elsewhere. The fact that no single machine can ever be reliable enough is particularly obvious when deploying servers onto a cloud deployment such as Amazon's EC2 service. Individual nodes are never expected to be as reliable as a traditional dedicated database server.

The architecture at cloud service provider Heroku solves this problem by reversing the normal approach to mission critical database deployment: they make the backups the most reliable part of the system. PostgreSQL's write-ahead log (WAL) mechanism can be used to make streaming, real-time backups of your database. The WAL-E tool developed there archives this data into Amazon's S3 storage, which so far boasts a perfect track record for storing all data ever committed to it. This approach is currently in use protecting over 100,000 Postgres databases, and has allowed Heroku to survive major interruptions in Amazon's cloud infrastructure with minimal data loss. In this session, learn how and why WAL-E was implemented, and how you can use it to keep your own data safe and secure.

There’s a lot of talk in the devops world about bringing developer concepts to system administration, and discussion the other way about bringing the awareness of operations to developers, but a lot of the conversation leaves out what is often the most critical part of your technology stack: the database. Perhaps that’s because DBA’s have always had to keep one foot in development and one in production, before there was a devops. Or maybe DBA’s just suck at playing well with others. Bottom line; it doesn’t matter. If you are going to store data, you need a plan that both developers and operations people can understand and embrace.

At OmniTI we’ve worked with many of the leaders in the devops movement and we’ve found there are commonalties across these organizations. It’s not so much about the tools, but about the techniques they use to help people break down barriers between different roles and establish a common ownership of technology within their organizations.

Monitoring and visibility, managing schema changes and production pushes, optimization, configuration and backups; there are aspects to data storage that bring about unique challenges. You won’t need to adopt all of these techniques to be successful, but it’s time you had a frank conversation about what it takes to make your database truly “webscale”.

Unscheduled

There are many strategies in scaling PostgreSQL infrastructure. Using Skytools, a set of technologies developed by Skype, Postgres can scale easily both horizontally and vertically. Learn in a hands on tutorial on how pgBouncer can be a significant boost to PostgreSQL performance and how to horizontally scale your PostgreSQL data with plProxy.

Skytools includes a variety of tools useful for scaling large PostgreSQL sites. The pgBouncer connection pooler allows incoming database load to be carefully controlled, as well as adding features for easier management of 24x7 availability. And the Skytools pl/Proxy language allows horizontal partitioning of data by ID across multiple servers, allowing a sharding style approach to scaling PostgreSQL. Learn how these tools have been used to scale upwards a PostgreSQL backed web site supporting billions of page views per month.