rsync (Blog)http://www.turnkeylinux.org/blog/term/57/0
enRsync the entire TurnKey library from a mirror close to you in under 5 minutes!http://www.turnkeylinux.org/blog/fast-rsync-mirrors
<p>
Like TurnKey so much you want a local copy of all the appliances but too lazy to download individual appliance images from SourceForge by hand via browser?</p>
<p>
I know exactly how you feel. Sloth <em>is</em> a virtue, and in the beginning was the command line.</p>
<p>
So now you can use rsync or ftp to batch download the entire virtual appliance library in whatever build type you like best from a <a class="reference external" href="http://www.turnkeylinux.org/mirrors">high-speed mirror</a> near you. The way the net gods intended!</p>
<!--break-->
<p>
Thanks to generous donations of bandwidth and storage space from opensource friendly network samurais around the world TurnKey now has 16 high-speed rsync/ftp capable mirrors in 12 countries: China, Ireland, United Kingdom, France, Germany, Sweden, Japan, Belarus, Bulgaria, Denmark, Argentina, and Israel. And we&#39;re just getting started...</p>
<p>
<a href="http://www.turnkeylinux.org/files/images/blog/network2_0.jpg"><img alt="" src="http://www.turnkeylinux.org/files/images/blog/network2_0.jpg" style="width: 650px; height: 433px;" /></a></p>
<p>
This means if your network is good enough you can now grab a copy of the entire TurnKey appliance library in any of the <a class="reference external" href="http://www.turnkeylinux.org/docs/builds">6 supported build types</a> (ISO, VMDK, OVF, Xen, OpenVZ and OpenStack) from a high-speed (e.g, 10-Gbps) mirror close to you in under 5 minutes.</p>
<p>
A few minutes ago in a practice run I rsynced TurnKey images to my Amazon EC2 instance at an average 288 Mbps <em>effective</em> download rate over the network.</p>
<p>
Wowsers! Up until I upgraded to a nice SSD a couple of weeks ago that was about as fast as I could copy files on my local hard drive.</p>
<p>
If you find these network speeds hard to believe I invite you to log into your TurnKey Hub account and launch a small instance in Ireland. Now let&#39;s rsync the entire TurnKey appliance library in Xen format - all 8 GBs worth from the HEAnet mirror:</p>
<pre class="literal-block">
$ rsync rsync://ftp.heanet.ie/[snip]/turnkeylinux/
Welcome to the HEAnet mirror site, ftp.heanet.ie (<a href="http://ftp.heanet.ie/about" title="http://ftp.heanet.ie/about">http://ftp.heanet.ie/about</a>)
-----------------------------------------------------------------------------
NOTE: All connections and transfers are logged; if this is disagreeable,
please disconnect now.
* ftp.heanet.ie is located in Dublin, Ireland and operated by HEAnet, the
Irish National Research and Education Network.
* This is a four node cluster with 10 Gigabit access to the HEAnet backbone.
* Please contact <a href="mailto:mirrors@heanet.ie">mirrors@heanet.ie</a> with any operational queries.
* You are connected to ftp-node2 (kokapetl)
-----------------------------------------------------------------------------
drwxr-xr-x 157 2012/02/12 17:05:27 .
drwxr-xr-x 4709 2012/02/08 18:00:08 iso
drwxr-xr-x 5686 2012/02/08 18:00:58 openstack
drwxr-xr-x 5434 2012/02/08 18:01:10 openvz
drwxr-xr-x 4930 2012/02/08 18:01:21 ovf
drwxr-xr-x 2759 2012/01/13 08:31:59 pve
drwxr-xr-x 5014 2012/02/08 18:00:44 vmdk
drwxr-xr-x 5266 2012/02/09 20:41:48 xen
$ time rsync -av -P rsync://ftp.heanet.ie/[snip]/turnkeylinux/xen ./
receiving incremental file list
xen/
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2
299570436 100% 30.43MB/s 0:00:09 (xfer#1, to-check=83/85)
xen/turnkey-appengine-11.3-lucid-x86-xen.tar.bz2.sig
490 100% 0.38kB/s 0:00:01 (xfer#2, to-check=82/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2
191193216 100% 34.68MB/s 0:00:05 (xfer#3, to-check=81/85)
xen/turnkey-bugzilla-11.3-lucid-x86-xen.tar.bz2.sig
490 100% 0.48kB/s 0:00:00 (xfer#4, to-check=80/85)
[ .. snip .. ]
sent 1629 bytes received 7993104501 bytes 36919658.80 bytes/sec
total size is 7992120707 speedup is 1.00
real 3m35.152s
user 0m29.290s
sys 0m30.220s
$
</pre>
<p>
Speaking with the authority of someone that used to download shareware from a local BBS 20,000 times slower (I.e., 6MB/hour on a 14400 baud modem) I can unequivocally state that this is just friggin awesome. The future is now!</p>
<p>
Shouts out to TurnKey mirror best buddies all over the globe:</p>
<ul class="simple">
<li>
Zhang from the USTC Linux User Group in China</li>
<li>
James from Bytemark hosting in the UK</li>
<li>
Arnoud from LIP6 in France, and Manuel from Ircam, also in France</li>
<li>
Carsten from RWTH Aachen in Germany</li>
<li>
Mattias from Umea Uni in Sweden</li>
<li>
Mitry from Beltelecom MGTS in Belarus</li>
<li>
Boian from IPACCT in Bulgaria</li>
<li>
Georg from dotsrc.org in Denmark</li>
<li>
Ariel from Cooperativa Telefonica in Argentina</li>
<li>
Lior from the Israel Internet Association in Israel</li>
</ul>
<p>
Not yet a TurnKey mirror best buddy but thinking you might want to be? If you have the resources to provide a mirror for TurnKey in your country reach out to our <a class="reference external" href="http://www.turnkeylinux.org/mirrors/new">global mirror commando team</a> 24 hours a day in any real or fictional language!</p>http://www.turnkeylinux.org/blog/fast-rsync-mirrors#commentsftpmirrorsnewsrsyncWed, 22 Feb 2012 05:51:39 +0000Liraz Siri3175 at http://www.turnkeylinux.orgUsing git and rsync to synchronize changes on a staging box to a live serverhttp://www.turnkeylinux.org/blog/website-synchronization
<p><em>The problem: working on a live web site is a bad idea</em></p>
<p>Anyone who's ever worked on a sufficiently complex web site knows it's a bad idea to work directly on the live server hosting the site for a couple of important reasons:</p>
<ol>
<li><strong>It's disruptive to visitors</strong>: If - sorry <em>when</em> you break something - your visitors are going to be exposed to it. Nothing creates a bad impression faster than a broken web site.</li>
<li><strong>Fear is stressful, stress kills productivity</strong>: you know if you mess around too much with the web site there's a good chance you'll break it. Naturally you don't want this to happen so your mind becomes preoccupied with the fear of making mistakes, and its hard to focus on what needs to be done.</li>
</ol>
<p>We develop this web site and test all non-trivial changes in a local <a href="http://www.turnkeylinux.org/drupal6">TurnKey Drupal</a> instance running inside a virtual machine. This means we can experiment and screw things up with no consequences. I find removing that source of stress makes you much happier and more productive as a web developer.</p>
<p>Working like this raises a few practical questions though:</p>
<ul>
<li>How do you push changes from the development box used for staging to the live web site without accidentally overwriting changes made by someone else?</li>
<li>How do you track who changed what?</li>
<li>When you screw things up on your development box, how do you reset the changes you've made and start again?</li>
</ul>
<p>&nbsp;</p>
<!--break-->
<p>When we started we didn't give that much thought to these issues and would just rsync a bunch of random directories around in ad-hoc fashion. That inevitably led to a few nasty mistakes, which convinced us this is something we needed to think through.</p>
<h2>Our solution: volatile-pull, volatile-sync</h2>
<p>Here's what we came up with:</p>
<ul>
<li>
<p><strong>One volatile directory to rule them all</strong>: we moved directories that were being changed (e.g., theme, modules) to /volatile and created symlinks from their original, sporadic locations on the filesystem.</p>
<p>A single volatile directory was much easier to keep track of mentally than an ad-hoc collection of directories.</p>
</li>
<li>
<p><strong>Revision control</strong>: on our development instances, we enabled revision control on /volatile by turning it into a Git repository with two main branches:</p>
<ol>
<li>'local' branch: where we committed our changes.</li>
<li>'remote' branch: contains a representation of the live /volatile</li>
</ol>
</li>
</ul>
<p>We then wrote a simple script (<a href="http://www.turnkeylinux.org/files/archive/volatile-sync.tar.gz">download</a>) which supports two operations:</p>
<ol type="A">
<li><strong>volatile-pull</strong>: pulls changes from the live server to the development instance
<ol>
<li>rsyncs the live /volatile to the 'remote' branch and commits.</li>
<li>merge the 'local' branch with the 'remote' branch while allowing the developer to resolve any conflicts between their own changes and the changes made by another developer.</li>
</ol>
</li>
<li><strong>volatile-sync</strong>: synchronize the development instance with the live server
<ol>
<li>call volatile-pull: before pushing our changes to the live server we always pull to minimize the risk we will accidentally overwrite changes made by another developer since we last pulled.</li>
<li>rsync the contents of the 'local' branch to the live server's /volatile</li>
</ol>
</li>
</ol>
<p>This technique should generally work for many similar development scenarios, not just Drupal web site development.</p>
<p>Note that pulling just before we push does not absolutely eliminate the risk of accidentally overwriting changes made by another developer, if they happen to be pushing at exactly the same time. To prevent this edge case you would need to implement some sort of locking mechanism. We didn't bother.</p>
<p><strong>Download</strong>: <a href="http://www.turnkeylinux.org/files/archive/volatile-sync.tar.gz">volatile-sync.tar.gz</a></p>
<h2>Database synchronization considerations</h2>
<p>With Drupal an extra caveat is that much of the web site lives in the database, so filesystem-level synchronization only solves a part of the problem.</p>
<p>Since the state of a live dynamic web site can change while you are working on the development instance (e.g., new users, new forum posts) we only pull the database from the live web site server to the development instance. Never in the other direction.</p>
<p>We only made an exception when we were upgrading the web site from Drupal 5 to Drupal 6. That required a massive database update we felt more comfortable preparing on the development instance and pushing out to the new Drupal 6 based live web site. In the meantime, we put a notice in the template of the old Drupal 5 site warning users that any change to the web site would be lost. 15 minutes later, the new Drupal 6 base site was up.</p>
<p><strong>Do you use a staging box? How do you synchronize? Don't hog your knowledge, <a href="http://www.turnkeylinux.org/blog/website-synchronization#comment-form">leave a comment</a>!</strong></p>http://www.turnkeylinux.org/blog/website-synchronization#commentsdevelopmentgitrsyncstagingThu, 14 Jan 2010 10:36:28 +0000Liraz Siri934 at http://www.turnkeylinux.org