Backup your Mac with rsync

Fortunately, Mac Leopard users have a program called Time Machine that makes things a lot easier. But is Time Machine the perfect backup solution? I don’t think so. There are a couple of things that make Time Machine very unsuitable for me:

You need to get a seperate external hard drive that can only be used for Time Machine (and has to be formatted first)

That drive has to be formatted in HFS+, hence, without any (commercial) third-party plugins it’s not readable on Windows or Linux systems

You have to leave your drive on all the time to make sure Time Machine makes backups

You can’t make a list of things you want to have backed up, you can only exclude folders from your complete hard disk

Time Machine makes an exact copy of your hard drive

Especially that last ‘feature’ is very irritating to me. I have an external drive with about 300G of files, including lots of music and video files. My MacBook drive is only 80GB big, so i can never have the complete contents of my external drive on my MacBook. Let’s say i have 10GB of MP3 files, which i backup with Time Machine, then i remove about 5GB of files from my MacBook to free some space. What happens when the next backup round is happening? Exactly, the 5GB of files get deleted from the external disk as well. When i want to play a certain MP3 file from my external drive i now have to ‘restore’ and ‘look back in history’ to find it. Not very user-friendly.

Luckily, there is a very good (free) alternative to Time Machine that does exactly what i want with backups: it lets you specify which folders you want to backup, it doesn’t delete things on the backup drive when you delete files from your original drive, and it’s compatible with any external drive and can even backup files over a network. This piece of software is called rsync. Here’s how to use it.

rsync is a command-line utility shipped with every copy of Mac OS X. It originated from the UNIX/Linux world, where it has been part of most Linux distributions for many years. rsync is reliable, fast, and easily configurable. Try running it by opening up the Terminal.app (located in your Applications/Utilities folder) and running the command:

rsync

You’ll get an overview of all possible options. In essence the syntax is very simple:

rsync OPTIONS SOURCE DESTINATION

What you’ll probably want is a one-way transfer of all files in SOURCE to DESTINATION, where only files are copied that are not available on the DESTINATION disk or different. Aside from that you’ll want to include all subdirectories, links, permissions, date/time, groups, owner and devices. To do that simply use this easy-to-remember option list:

rsync -rlptgoD

Ha, just kidding! Fortunately there is another switch that does all of that with one switch, namely the archive switch:

rsync -a

So, let’s say you want to backup the files in your Documents directory to your external harddrive, which you appropriately named ‘backup’, then this would be the command:

rsync -a ~/Documents/ /Volumes/backup/Documents

For those of you who don’t use the Terminal very often: the tilde (~) is a shortcut for your home directory. If, for example, your name would be ‘Alice’ your home directory would probably be

What is rsync?

Rsync is a program for synchronizing two directory trees across different file systems even if they are on different computers. It can run its host to host communications over ssh to keep things secure and to provide key based authentication. If a file is already present in the target and is the same as on the source the file will not be transmitted. If the file on the target is different than the one on the source then only the parts of it that are different are transferred. These features greatly increase the performance of rsync over a network.

What are hard links?

Hard links are similar to symlinks. They are normally created using the ln command but without the -s switch. A hard link is when two file entries point to the same inode and disk blocks. Unlike symlinks there isn’t a file and a pointer to the file but rather two links to the same file. If you delete either entry the other will remain and will still contain the data. Here is an example of both:

Why backup with rsync instead of something else?

Disk based: Rsync is a disk based backup system. It doesn’t use tapes which are too slow to backup (and more importantly restore) modern systems with large hard drives. Also, disk based backup solutions are much cheaper than equivalently sized tape backup systems.

Fast: Rsync only backs up what has changed since the last backup. It NEVER has to repeat the full backup unlike most other systems that have monthly/weekly/daily differential configurations.

Less work for the backup client: Most of the work in rsync backups including the rotation process is done on the backup server which is usually dedicated to doing backups. This means that the client system being backed up is not hit with as much load as with some other backup programs. The load can also be tailored to your particular needs through several rsync options and backup system design decisions.

Fastest restores possible: If you just need to restore a single file or set of files it is as simple as a cp or scp command. Restoring an entire file system is just a reverse of the backup procedure. Restoring an entire system is a bit long but is less work than backup systems that require you to reinstall your OS first and about the same as other manual backup systems like dump or tar.

Only one restore needed: Even though each backup is an incremental they are all accessible as full backups. This means you only restore the backup you want instead of restoring a full and an incremental or a monthly followed by a weekly followed by a daily.

Cross Platform: Rsync can backup and recover anything that can run rsync. I have used it to backup Linux, Windows, DOS, OpenBSD, Solaris, and even ancient SunOS 4 systems. The only limitation is that the file system that the backups are stored on must support all of the file metadata that the file systems containing files to be backed up supports. In other words if you were to use a vfat file system for your backups you would not be able to preserve file ownership when backing up an ext3 file system. If this is a problem for you try looking into rdiff-backup.

Cheap: It doesn’t seem like it would be cheap to have enough disk space for 2 copies of everything and then some but it is. With tape drives you have to choose between a cheap drive with expensive tapes or an expensive drive with cheap tapes. In a hard drive based system you just buy cheap hard drives and use RAID to tie them together. My current backup server uses two 500GB IDE drives in a software RAID-0 configuration for a total of 1TB for about $100 which is about 1/6th what I paid for the DDS3 tape drive that I used to use and that doesn’t even include the tapes that cost about $10/12GB.

Internet: Since rsync can run over ssh and only transfers what has changed it is perfect for backing up things across the internet. This is perfect for backing up and updating a web site at a web hosting company or even a co-located server. Internet based backup systems are also becoming more and more popular. Rsync is the perfect tool to backup to such services over the internet.

Do-it-yourself: There are FOSS backup packages out now that use rsync as their back end but the nice thing here is that you are using standard command line tools (rsync, ssh, rm) so you can engineer your own backup system that will do EXACTLY what you want and you don’t need a special tool to restore.

As a key enabler for software-defined networking (SDN), OpenFlow was initially introduced in the academia as a way to enable innovation on production networks which had traditionally been built with closed and proprietary networking hardware. OpenFlow offloads the high-level routing/forwarding decisions (control plane) from networking devices such as switches, and moves the control plane on to a separate controller. The networking devices then simply forward traffic, as programmed by the external OpenFlow controller. It is the OpenFlow protocol that is used by the OpenFlow controller to program the networking devices.

Suppose you have an OpenFlow testbed running, which consists of an OpenFlow controller and a set of OpenFlow-capable switches. For troubleshooting purposes, you want to capture and examine OpenFlow messages exchanged between the controller and the switches. For this you could monitor exchanged OpenFlow messages either at the controller or the switch side, but what if it is not convenient to do so? Another way is to « sniff » network packets on the OpenFlow control channel and interpret the packets.

In this tutorial, I am going to show how to sniff live OpenFlow control packets and decode OpenFlow messages contained in the packets.

Note that for such packet sniffing to work, SSL must be disabled in any existing OpenFlow control channels between the controller and switches. Let’s assume we are not talking about any production environment here, so the SSL is off for now.

Method One: Sniff OpenFlow Messages via Wireshark GUI

If you want to monitor OpenFlow messages using packet sniffing, the most user-friendly way is via Wireshark, a GUI-based packet sniffer. A nice thing about Wireshark is its extensive list of built-in and custom dissectors. Each dissector decodes some part of packet data based on a specific network protocol. For pretty much any existing network protocol, there is a corresponding Wireshark dissector (either built-in or contributed by a third-party). The OpenFlow protocol is not an exception.

While there is an official OpenFlow dissector, I am going to use a third-party OpenFlow dissector developed by Big Switch Networks, since the former seems to have patchy/incomplete support for different OpenFlow versions.

Suppose you as a software developer has set up daily builds of your software for testing purposes. Every day you make a new build, users have to re-download the updated build to evaluate it. In this case you may want to enable differential downloads, so that users can download only difference between two builds, thereby saving on the server’s bandwidth. Users will also be happy as they don’t have to wait to re-download the whole thing. Similar situations are encountered when you want to set up a download archive which allows incremental sync for users.

In these cases, how would you distribute incrementally updated files efficiently for multiple users? In fact, there are open-source storage solutions that come with « delta sync » capability built-in, such as ownCloud or Syncthing. These kinds of full-blown solutions with built-in GUI require users to install a dedicated client, and thus may be an overkill for simple file distribution that you are looking for.

Barring full-blown third-party software like these, perhaps rsync may come to mind, which can do bandwidth-efficient file sync. The problem with rsync, however, is that all the heavy duty computations is done at the server side. That is, when a client requests for a file sync, the server needs to perform block-by-block checksum computation and search for blocks not available on the client. Obviously this procedure can place a significant strain on the server’s resources if the server needs to handle many users, and thus is not scalable.

What is Zsync?

This is when a command-line tool called zsync comes in handy. While zsync uses the same delta-encoding based sync algorithm as rsync does, it moves the heavy duty computation away from the server and onto the clients. What do I mean by that?

Well, in zsync, the server maintains a separate .zsync metadata file for a file to distribute, which contains a list of « precomputed » checksums for individual blocks of the file. When zsync client requests for a file sync, the client downloads .zsync metadata file first, and performs block-by-block checksum calculation to find missing blocks on its own. The client then requests for missing blocks using HTTP range requests. As you can see, the server is totally out of the loop from the sync algorithm, and simply serves requested file blocks over HTTP, which makes it ideal when incremental file sync is needed for many users.

Here is a quick rundown on the server-side overhead difference between rsync and zsync. In the plot below, I compare rsync and zsync in terms of the server’s CPU usage when 200 users are downloading a tarball file with 2.5% discrepancy of a previous version. For fair comparison, SSH is not used for rsync.

With zsync, since all checksum computation overhead has shifted from the server to individual clients, the server overhead is reduced dramatically. A small neat idea makes zsync a real winner!

In the rest of the tutorial, I will show how to distribute a file incrementally using zsync under the Linux environment.

Zsync: Client Side Setup

On the client side, you need to install zsync to initiate file transfer from a remote web server. zsync is extremely lightweight, and is included in the most Linux distros. Here is how to install zsync on various distros.

Arch Linux:

When a group of employees collaborate on a project, there is need for handling all aspects and complexities of the project from start to finish, such as task assignment, scheduling, budget management, communication, documentation, and so on. Thus a good project management tool is a necessity for any kinds of businesses or organizations which are involved in a lot of collaborative projects.

Especially web-based online project management tools are the top choice for such environments as different project members can access and share the online tools wherever they are. If your organization requires good web-based project management software, you will find ONLYOFFICE a viable candidate.

ONLYOFFICE is a free and open source office suite that enables seamless project collaboration in the web-based environment. It offers different management tools for documents, emails, events, tasks and client relations. Using ONLYOFFICE, a project manager can schedule the workflow, lead discussions, monitor and coordinate the project progress using Gantt Chart and reports.

In this tutorial, I will describe the main project management features offered by ONLYOFFICE and give some tips to use them more efficiently.

Installing ONLYOFFICE using Docker on Linux

The easiest way to install ONLYOFFICE is by using Docker. The tutorial presented here covers the Docker-based installation instructions for the whole ONLYOFFICE system with online editors integrated for real-time document collaboration.

If you want to install ONLYOFFICE without editors (called « community server »), simply execute the following command:

The above command will download the official ONLYOFFICE Docker image with all dependencies needed for its correct work. For more installation and configuration tips, please refer to the ONLYOFFICE official documentation.

Running a Self-Hosted Web Office

To open your web office, enter in the address bar of your browser localhost (http://IP-Address/) on which the docker is running. The Welcome page will open:

Enter a password and specify the email address you will use to access your office the next time.

Getting Started with a New Project

First, click the Projects link to open the project management tool. To start a new project, click on the Create New…button in the upper left corner and select the Project option from the drop-down list. Fill out the form: specify your project Title and enter a short Description. If necessary, assign a Project Manager and create a project Team.

To restrict access to the newly created project, check the Save this project as private box. Review the project settings and click on the Save button.

Note: to create a new portal, the full privilege administrator or projects management administrator rights are required.

Setting up Your Project Hierarchy

Once the project is started, the next step is to create the project hierarchy which involves the following:

Set up project key points with milestones

Distribute the whole amount of work among the team members using tasks

Link tasks to determine the order in which they need to be performed

Divide a task into subtasks if it requires several persons to perform several steps

Organize Project Documentation

Keep all the documentation related to your project in one place to access it anytime anywhere. Switch to theDocuments tab on your project page to:

Upload documents from your hard disk drive

Download an existing document to your hard disk drive

Filter and sort documents to facilitate the search

Copy or move documents to another folder

Create a new document, spreadsheet or presentation, open and edit the existing ones if the online editorsare installed

Attach documents to the related tasks

Managing Access Rights

Being a project manager you can manage the project team, tasks, milestones, discussions, comments inside your project; change project status, edit its settings, and also set access rights to project team members for a private project.

By default, each member of your project team can add tasks, assign them to himself or other team members and manage them; start new discussions; add and remove documents; view all content inside your project: milestones, tasks, documents, discussions and contacts.

To restrict the access to an item, switch to the Team tab on your project page, select the team member you wish to set access rights for, and click on one of the links to the right: Discussions, Documents, All Tasks, Milestones,Contacts.

Note: you can set the access rights for each team member unless he/she does not have full privilege administrator or projects management administrator rights.

Monitoring Project using Gantt Chart

Making changes to your project once it is underway is both indispensable and inevitable part of its life cycle. The simplest way to do it with ONLYOFFICE, especially when you manage multiple related projects, is to use Gantt Chart, a bar chart used to illustrate a project schedule.

To open Gantt Chart, click on the Gantt Chart link on the left side panel, or click on the Gantt Chart button in the upper right corner of your project page.

Gantt Chart gives you a panoramic view of all your project activities and allows you to:

Useful Tips

To find a solution of problems occurred, you can start a project-related discussion and discuss the issue with your project team within the Discussion tab.

2. Using templates

To easily start a new project, you can use the previously created templates (click on the Create New… button and select the Project Template option to create one).

3. Reporting

To keep up with what each team member is doing, make use of the Reports system. Click on the corresponding link on the left-side panel and generate a report or set up needed parameters once and regularly receive automatic reports by email on certain date and time.

When disaster strikes (and it will), you can put the pieces back together if you’ve got a solid backup solution in place.

If you have a Windows server and you’re not backing it up, you are tempting fate. It’s only a matter of time before something tragic will happen: a hardware failure, a hacker, or a corrupted Windows update. Something will bring that server down and do everything it can to take your data with it.

Luckily, several reliable backup solutions are available for the Windows Server environment. They aren’t perfect. But the five solutions discussed here have their strengths and can be depended upon to keep your data safe. Now, before I reveal the big five, I want to make sure you understand how critical it is to be backing up that data to an external drive and to rotate that drive offsite. If you’re backing up drives and leaving everything onsite, should a fire (or some other catastrophe) occur, that data will be lost.

1: Acronis Backup and Restore (with Universal Restore)

Acronis Backup and Restore (Figure A) is the backup tool I use the most. It’s costly but well worth it. The newer version of Acronis does a great job of ensuring that there is room on your external drive for your latest backup. You can overwrite the old backup or go by the traditional cleanup rules established in Acronis Echo. But if you go the Acronis route, you should also purchase the Universal Restore feature. With this tool, you can take a full image of a backup and restore it to any hardware — so long as it meets the requirements of your platform. That way, you don’t need an exact duplicate of your server hardware sitting around waiting to have an image restored onto it. Acronis also has add-ons for Exchange and SQL Servers.

Figure A

Acronis Backup and Restore

2: Symantec Backup Exec

Symantec Backup Exec (Figure B) is a powerful backup solution that does a great job of « just working. » And with its at-a-glance calendar, it’s easy to know when your backup has succeeded and when it has worked. If you’re using Symantec Backup Exec across a network (to back up clients/hosts), you can feel safe because it uses SSL for all network backup connections. Like Acronis, Symantec offers solutions for Exchange and SQL. Also, the ability to manage Vaults (backup destinations) is superb with Symantec. Just make sure that if a Vault is physically (and permanently) removed, you remove it from within the backup software. Otherwise, it will cause problems.

Figure B

Symantec Backup Exec

3: Carbonite

Carbonite (Figure C) is primarily an online backup (although the Business and BusinessPremier plans allow you to back up to an external drive). It’s meant to back up data only. Carbonite is probably one of the most reliable online backup tools and can back up email, documents, POS files, financial data, pictures and other multimedia, and more. But to use Carbonite for Windows Server, you must use the BusinessPremier Plan ($229.00 per year), which will allow you to include as many servers running Window Server 2003/2008/2011 as you like. Unlike Acronis, you won’t be backing up an image of your server with Carbonite — it’s a data-only backup tool. But it’s an excellent addition to any onsite backup solution.

Figure C

Carbonite

4: Windows NT Backup

Windows NT Backup (Figure D) is the free backup solution that comes with any Windows Server. Why do I include it here? Although it just does data backup and doesn’t offer many of the bells and whistles of the other tools, it is one of the only truly reliable backup solutions that will clear the Exchange logs for you. I always set up an NT Backup to run once a week (or month, depending upon the needs), just to have it clear the Exchange logs. Otherwise, you’ll be including a lot of unnecessary data in your backups, causing their size to get out of hand.

Figure D

Windows NT Backup

5: ShadowProtect 4 (Server)

ShadowProtect 4 (Figure E) is probably the least-known product in this list. But it falls into place next to Acronis as a feature-rich tool that can do everything — including recover to dissimilar hardware. ShadowProtect 4 also includes an easy-to-use tool that allows you to consolidate backup image files and convert the backups to either .VHD or .VMDK virtual machines. Another outstanding feature is its granular recovery, which allows for the quick recovery of individual files from within a backup. Unique to ShadowProtect is the Virtual Boot technology, which provides for automatic failover to a virtual server.

Figure E

ShadowProtect 4

Which backup?

Every company needs a backup plan. How far you go with your backup depends upon the size of the company, your needs, and your budget. But any of the above solutions will serve as a great place to start — or stop — for your Windows Server backup needs. Do you have a different backup solution to add to the list? Share your recommendations with fellow TechRepublic members.

Minimum Word Count

Do you run a blog with multiple authors? Then, you probably have wondered how you can set a minimum word count for your posts in WordPress. In this article, we will share with you a snippet that lets you set a minimum Word count for your WordPress posts. If a user tries to publish a post that is too small, then it will return an error telling them the post is not long enough.

Simply open your theme’s functions.php file and paste the following code: