Month: December 2011

We’ve made great progress on the new repository and web site servers and are happy to report that all is going very well! As you may recall, we’ve been having server problems for some time on our 5 year old hardware and have been undergoing the first phase of our modernization process! Time for another update…

We split the FreePBX servers into two virtual systems that represent this website and the Module Admin online repository that has often been referred to as the mirror despite the fact that there was only one such site. We have spent the last couple weeks staging these systems and migrating the environment to create an exact replica of the current FreePBX.org server with a handful of necessary changes to improve it. During the last week, we’ve been syncing the live site to these two virtual systems every two hours to assure a completely up-to-date environment. In addition to the webstite and repository we also have an identical replica of both these systems that will be part of the high availability setup.

Yesterday we took some new steps. We added the live SQL replication between the new website and its backup as well as the same for the repository system. Currently both the primary and backup systems live on the same physical server but that will be changing within the next couple weeks as we are preparing a home in a second facility where these two alternate systems will live. In the case of the repository we have built in an automatic failover ability in FreePBX 2.10 and will be enabling that ability shortly. As always, Schmooze Com Inc. has been invaluable in their technical expertise in helping us prepare for and stage much of this and a special thanks goes to Tony Lewis for much of this!

The second big step we took yesterday was to make the repository live by pointing the DNS record to the new server. We’ve watched the traffic to the old repository die to almost a trickle as badly behaved DNS sources have slowly updated their stale records. (We had set the TTL to 60 in preparation for this). So far we have not seen any concerns raised about this migration.

If all goes well, we plan on bringing the new website online tonight by pointing the DNS record to it. For most of you this should be a seamless transition that goes unnoticed. For those of you who receive stale DNS data that there will be glitches. This is often the case for OpenDNS and Microsoft based DNS servers. If you fall in one of these categories then you will end up on a web page that gives you further instructions on how to get to the site. (If your timing is bad, you may hit a very short window of about 5-10 minutes when we disable apache to get the last sync finished and assure there is no data loss from bad timing.)

Just a quick note on the upcoming FreePBX OTTS Class. The current Early Bird Special expires Friday so if you are thinking of coming, don’t let yourself get surprised. We are also well over half full at this point and are looking forward to seeing some of you next month!

For now we’ll update you as soon as we have more news on all that is going on!

With the holidays fast approaching we’re firing on all pistons trying to move everything forward, keep the site healthy and prepare for those of you who will be [url=/open-telephony-training-seminar]joining us in Naples, FL for the next Open Telephony Training.![/url] Progress is solid on the new repository site and website that we discussed [url=/news/2011-12-03/server-crashes-up-coming-trainings]in the last blog.[/url] and version 2.10 is close to entering release candidate state.

On [url=/open-telephony-training-seminar] the OTTS training[/url] space is filling quicker than we expected so don’t procrastinate. The current [url=/open-telephony-training-seminar-register]Early Bird Pricing[/url] runs out in just a few days so take advantage of the great discount and book now. We hope to get to meet you next month and get your feedback on 2.10!

Concerning the next release, we are wrapping up many of the loose ends and will be publishing the version 2.9 to version 2.10 upgrade module within the next week or so, which will allow many more users to help test the waters on this great release. If the feedback goes well it will give us the green light to move 2.10 into the release candidate state for the final testing and push to a final release. Please hop on board and tell us what you think so we can assure a great release that meets our user community expectations!

Concerning the [url=/news/2011-12-03/server-crashes-up-coming-trainings] server troubles we mentioned last blog,[/url] great progress is being made. At this point the new server is being delivered to the datacenter tomorrow. We have staged the new virtual machine containers for the main website (this blog and all that surounds it), and for the repository site as well as a backup repository site. We are currently replicating the main site to these virtual machines every 2 hours to assure we are protected were disaster to hit prematurely. Once the server is installed in the data center we will move the virtual instances onto the new hardware and then begin the migration!

If all goes well you may be already reading this blog on the new site! We are shooting for a cut-over this weekend barring any unexpected glitches. if that is a bit aggressive, since we do like to set lofty goals, we should at least be able to get the repository mirrors up this weekend. Concerning the repository site replication for availability, we’ve decided to update FreePBX to handle multiple mirror sites. It will always try the primary site but fail over to any alternate sites that are configured. We will implement this in 2.10 and beyond, and for 2.9 and earlier releases we will handle failures by pointing the DNS record to the mirrored server in the case of failures which will address most systems.

For now that’s what we have going, we’ll keep you up-to-date on the server plans as we move forward!

If you spend much time here or if you have a lot of FreePBX servers that you look after then you are probably aware of the troubles we have been haunted with over the last months. We want to let you know more about this and what we are doing to address these issues and assure there is a robust infrastructure going forward. In the mean time, let me remind those of you about our [url=/open-telephony-training-seminar]Fast Approaching FreePBX Training[/url] that we told you about in our [url=/news/2011-11-07/2-10-beta2-and-come-see-us-in-naples-florida]last blog[/url]. There’s still time to take advantage of the [url=/open-telephony-training-seminar-register]Early Bird Discount[/url] that won’t be around much longer so act fast. Also keep in mind we had to turn away multiple attendees in our last training so register before it fills up! For those of you that take the expense off in your taxes, act quick to get a 2011 deduction!

The last time we gave our infrastructure a boost was almost 5 years ago in early 2007 when we were having problems with the old hosting facility and underpowered hardware. We’ve been chugging on that same piece of iron now and it’s starting to show. The site was down for several hours the day after Thanksgiving which was eventually diagnosed to a bad NIC (network interface). It went down again a few days ago for what looked like a disk starting to go bad. Beyond these, we’ve had problems for months, often with the system thrashing on swap space. Despite reasonable efforts trying to locate a root cause we eventually resorted to a nightly restart of apache to help temper the system, a real [i]”ghetto”[/i] bandaid we are not proud of.

The time has come to deal with our infrastructure head on so here’s a little information about what makes up the FreePBX site and the steps we are taking to assure a solid future. We will approach this in two phases, the first and most urgent being to take the current environment and move it to new hardware and facilities along with some additional changes to make it more robust. The second and longer term step being to investigate the modernization of the core underlying applications that make up the FreePBX site.

At a macro level the FreePBX site is made up of two primary components. There is this website you are currently reading which uses a dated version of the [i]Drupal Content Management System (CMS)[/i] in conjunction with a dated version of the [i]Trac[/i] ticketing/project management system paired with the [i]Subversion (SVN)[/i] repository. There are a handful of other components and some custom modules as well that glue everything together and provide the single sign-on ability between Drupal, Trac and the underlying SVN access. The second component is the FreePBX [i]mirror[/i] site which provides the backend infrastructure for the online Module Admin repository and server side publishing tools as well as the general repository for all other FreePBX tarballs and downloads. This software is almost all custom and is used to manage the environment in conjunction with a little MySQL There is actually a third component that we have not touched on, which is all the parts involved in the FreePBX Distro. This was developed from inception on a new infrastructure so does not need to be addressed here. In essence the old FreePBX infrastructure is now catching up with the lead that the Distro has taken.

With a lot of help from Schmoozecom.com, we have taken some initial steps already which include securing new and much beefier hardware at a new hosting facility, and staging the the [i]Site[/i] and [i]Mirror[/i] into two virtual instances that will be placed on the new hardware. Our intention is to replicate the Mirror site to a second hosting facility and enhance the backend environment to keep both of these up-to-date at all times. We are currently considering either roundrobin DNS records for some level of high availability and load balancing on the Mirror sites, or updating FreePBX to do it directly. Both have their advantages and flaws. For the Site configuration, we plan on having a duplicate virtual instance which will be constantly replicated from the main Site using a master-slave MySQL replication in conjunction with [i]rsync[/i] and other custom scripts to assure an up-to-date copy at all times. We will also be employing other levels of traditional backup for both to assure that everything is robust. We want to re-iterate the gratitude to Schmooze, and in particular to Bryan Walters (GameGamer43) and Tony Lewis for their dedication as well as expertise in helping with this transition.

At this point we are already in process of creating these two environments and we hope to do an actual switch over in the next two weeks. If all goes well it should be almost transparent though we will update you when this will be happening and of any planned outages we may need just prior to assure a clean transition with no data loss. Once we are up on the new infrastructure we will be well positioned to start planning the second phase which is the modernization of the Drupal and Trac environments. Since the new Site will now live within a virtual container, it will be very easy to replicate into a development environment that will allow us to work on a migration plan to updated the software, which has not been something we have had available up to now. We will provide more news on this process once we get through the first phase as well as let you know where community help might be desired in doing such.

A quick note on the 2.10 release. We continue to work strong on this with a handful of last minutes additions we have put into it. Juggling between the site issues and the beta release have drawn it out some though there are dozens of production sites that are running solidly on the 2.10 beta with Asterisk 1.8 so we are feeling very strongly about this. We plan on teaching the next [url=/open-telephony-training-seminar]FreePBX Open Telephony Training Seminar[/url] on 2.10 in sunny Naples, Florida next month so rest assured we have strong motivation to get things wrapped up with this release. We hope to see many of you next month so don’t procrastinate and end up having the class filled up on you! Get registered now, get your 2011 tax deduction and know you will have some sun in the middle of the winter next month!