September 1st 2009

Upgrading to the latest OpenOffice 3.x under Hardy Heron

Ubuntu does not automatically upgrade to the latest issue of OpenOffice, it just applies any updates to the version which came with the original Distribution ie 2.4 in the case of Hardy Heron. There are advantages in some cases in upgrading to version 3.x which is in Jaunty.

You can use the Synaptic Package Manager to update everything (Add/Remove was not happy with these changes until I had used Synaptic). System -> Administration -> Synaptic Package Manager

First update the package lists by clicking Reload

Then click Mark all Possible upgrades which should show all the ones for Open Office upgrade and click Apply.

You should now find that OpenOffice has been upgraded.

17 October 2009

Self Publishing and selling on the Web

A number of 'Services' are required if one is to self publish books and sell them.

A print on demand self-publishing site preferably with country specific versions but with online worldwide retailing - the service by which others are judged is Lulu.

A means to make and receive payments for internet transactions. Paypal is the best known service for individuals and is now widely used and is the service by which others are judged.

Email, preferably a separate account used only for the publishing and other sales activities as email address is also used as a login address for some services and/or is available to customers.

A full bank account which allows direct debit access to be set up and trans fers in by BACS, CHAPS etc transfers. It is best if this has internet access as some of the validation techniques need a rapid response. There are advantages if this is again a separate account to reduce the risks of internet fraud and also make accounting easier in the unlikely event one makes a lot of money.

The Publishing and Payment services both obviously have to be able to handle payments to you and be linked to a bank account so the proceeds can be passed to you. They need to have good and visible internal accounting with transactions details stored for a long period, preferably with the ability to do analysis.

Lulu

Lulu enables one to upload files in a number of formats as well as a create or upload a cover page some of the details have already been covered above but this section will expand that to the actual use of Lulu. Before you can start you have to create an account but even before that most of the help files are available.

Once you have understood exactly what you need the actual production of a book on Lulu is fairly straight forwards but you should have a practice run before you produce the final version as changes become more difficult as you reach the end stages of the process. Repeating the production process with existing tested files to get a clean copy for printing a first copy for printing then takes only a few minutes. Once you have finalised the book enough to print it become slightly more difficult to make changes, in effect you are then under a version control system and although it can be withdrawn and hidden you can never completely remove a book that has reached that stage. You should however always produce one or two physical copies before committing to a larger print run or taking the next stage and making it available to everyone via Lulu as there may be a few errors which are not obvious on the screen - we made two sets of changes following printing, the first was a few cases of number being use as say 9 when convention says it should be nine and after that correction was printed a we found that version had a slightly different size of page number in the appendices to the main body of the book. Neither would have been a cause to reprint or add a correction to a conventional book but print on demand allows such minor corrections to be made. However if you make anything more than the most minor correction after it is fully in print and your copies have been sent to the copyright libraries then a new ISBN is required.

If you are going to publish a book through Lulu which they distribute and which has an ISBN they impose some additional quality control as their name is on the line. In particular the final PDF file which is sent to the printer has to be generated by genuine Adobe software or by their own converter from other formats. This converter allows you to add together several files and merge them. We are using the OpenOffice word processor which outputs an acceptable for most printers but we have used there converter to put a 'stamp of approval' on it by appending a blank page from another file at the end - the blank page is required by most printers so this does not increase the length in practice. So, at the end of the day, we have to upload three files to Lulu as we work through their publication Wizard, The PDF of the books text, a file containing a blank page to terminate the book, force checking of the PDF and producing a PDF which is hopefully approved for any form of sales package or publisher and the last file is the cover, again in PDF format.

Money you earn can only be remitted by cheque or to a PayPal Account and this is set up by My Lulu -> My Accounts tab -> Change Remission Settings (on the left hand side menu)

Payments you make for books and services can be paid by credit card which can be changed or set up My Lulu -> My Accounts tab -> Credit Card Information (on the left hand side menu)

PayPal - Basics

Paypal is very popular way of making and receiving payments and money transfers through the Internet. PayPal serves as an electronic alternative to traditional paper methods such as checks and money orders. A PayPal account can be funded with an electronic debit from a bank account or by a credit card. The recipient of a PayPal transfer can either request a check from PayPal, establish their own PayPal account for the deposit or request a transfer to their bank account. PayPal is an example of a payment intermediary service that facilitates worldwide e-commerce. Whilst PayPal is a wholly owned subsidiary of eBay with corporate headquarters in the United States, it also operates as a Luxembourg-based bank with appropriate regulation within the EU. Currently, PayPal operates in 190 markets, and it manages over 184 million accounts, more than 73 million of them active. PayPal allows customers to send, receive, and hold funds in 19 currencies worldwide. It handles about £40 billion of transactions per year.

Each type of account has different features, including different sending and receiving limits. When you sign up on their site you are seemingly only offered the choice of Personal and Business accounts but that is because the personal and premier accounts now seem to have converged and in practice you seem to get an account which is able to receive payments - this account also allows you to use some of the 'Merchant' facilities which means you can set up for sales of one off items (rather than a full emarketing service with shopping basket etc) on you web site or by a link in an email.

PayPal Account verification

When you first set up the account everything is in place and useable but there are a number of restrictions on the amounts that you can receive or send per transaction or per year. These restrictions can then be lifted - the first and essential step is to validate the link with your nominated bank account. Validation is achieved by PayPal making a series of very small transfers into your account which you then have enter on the PayPal web site within a time limit. These transfers take several days to arrive so you need internet banking to keep checking or access to telephone banking or via an ATM. This raises the limits to around£1000 per year beyond which more verification is required because of money laundering etc regulations. You also need to fill in some information which they call business information but has the ability to handle an individual - this needs information on your expected methods of sales, average transaction sizes and monthly transaction estimates.

Selling on Paypal - PayPal Merchant Services

When starting up with a low level of transactions two of the Merchant services are appropriate - Email Payments and Website Payments Standard. Both are available form a Premium Account.

Email Payments

Email Payments allows you to accept payments by credit card, debit card and bank transfers all by email. You don't need a website – you just send an invoice or request for money from PayPal's website to your customer's email address. They just click on a link to pay you via PayPal. Your customers don't even need a PayPal account to pay you

Website Payments Standard

This Service allows you to use a tool on the PayPal web site to create a button for you web site which is linked to an payment form on the PayPal web site which allows customers to purchase using credit cards, debit cards or a PayPal account. You specify all the basic information on the PayPal web site such as the item name, price, postage etc. and the Wizard will produce the HTML code to copy onto your site. There is considerable customisation possible and you can, for example, use your own image for the button and specify two web pages to be transferred to following successful and unsuccessful translations. It is reached by Paypal Web Site -> Login -> Merchant Services -> Buy Now Buttons (at right of section headed Website Payments Standard).

The advanced options also allow you to personalise the payment web page on the PayPal site using some additional codes put into a section called Add Advanced Variables in the third (optional) step when creating a Buy Now Button.

image_url - The URL of the 150x50-pixel image displayed as your logo in the upper left corner of the PayPal checkout pages.

cpp_header_image - The image at the top left of the checkout page. The image’s maximum size is 750 pixels wide by 90 pixels high. PayPal recommends that you provide an image that is stored only on a secure (https) server.

cpp_ headerback_color - The background colour for the header of the checkout page supplied as a case-insensitive six-character HTML hexadecimal colour code in ASCII.

cpp_ headerborder_color - The border colour around the header of the checkout page. The border is a 2-pixel perimeter around the header space, which has a maximum size of 750 pixels wide by 90 pixels high. Valid value is case-insensitive six-character HTML hexadecimal colour code in ASCII.

6th November 2009

Edimax Wireless 3G Broadband Router 3G-6200n

I have been looking for a while for a way to get mobile broadband through a WiFi router so we can share an internet connection and networking whilst we are away from home which uses a USB Mobile Broadband dongle. One big advantage is the way these work is identical to any other ADSL, Firewall, WiFi router and are independent of operating system as they are set up and controlled by a web interface. This has been available for a while through 3 but it was locked to their system. Edimax, who I know little about, have brought out a box which seems to do almost everything I want and I have been trying it out. The only shortfall is that it runs off mains via the usual fat plug which supplies 12v at 1 amp to the box and there is no car adapter which would make it perfect for our narrowboat - I have discovered that the eeePC runs off 12volts 2 amps and there are lots of cheap adapters and all I will need to do is adapt the plug into the router.

I bought Edimax Wireless 3G Broadband Router 3G-6200n from Digital Components Ltd for £37 plus the usual extortionate postage and packing but that was amortised as part of a larger order. The actual box is quite light and compact (300gms) and likewise the mains adapter is one of the smallest I have seen. It comes with instructions which are quaint but can be understood and a full manual on CD along with a program to install it on Windoz if you do not want to use the web interface. You first need to connect via a network cable which they provide so you can set up the Wifi. This is easy and you connect via a web interface to 192.168.2.1 which is an excellent choice as most people will be using 192.168.1.1 for their normal router. This takes you into a login screen which displays the default username and password of admin:1234 and then to a Quick Setup which takes you through a subset of the setup I cover below for the Mobile Broadband Dongle and WiFi.

You first get a screen to allow you to enter your location, the other items are preset and fine. You can no plug in the Broadband Dongle if you have not already set it up and click 3G on the next screen. The next screen is where you set up the APN for your provider (pp.internet for Vodafone PAYG) and username and password (web web although I think anything will do) and the dial script (this is almost always *99#). You finally get to the Wifi Setup where you set up the SSID (edimax) channel which should be different to any other Wifi boxes to avoid interference (6).

You now need to setup security. I use WEP 128 bit although there are better systems as everybody can use it and add on Mac address filtering at a latter stage as that restricts access to particular machines only - the Mac address is unique and built into the network hardware.

You access the Security at a later stage via General -> Wireless to Security Settings. You need to select Encryption WEP Key length 128 Key format hex and enter a memorable 26 bit hex number (memorable is a joke) but by repeating a shorter number to make it up to 26 long you have a hope of recalling it. As with everything you need to click apply until you end up having to wait for 30 seconds while it set up the router hardware. If you have Linux machines or the latest Windows drivers you will be able to use WPA which is much more secure and also easier to set up as it has shorter passcodes and I am changing to it on my machines which will be away from home.

Mac access is setup by General -> Wireless -> Access Control. You need to know the Mac addresses of all your machines and add them. When you add them you do not include the : between each one although they are displayed with it! If you do not know them you can find out by Wireless -> Basic settings -> Show Active clients which conveniently stays open in a separate window so you can copy the - cut and paste does not work as they are displayed with the colons! Add them all especially the machine you are working on and then tick the box and Apply etc until you have a 30 second wait. If you get it wrong and cut yourself off you can always use the cable interface to correct your Mac address.

Not surprisingly the box seems to work fine for networking between machines and the internet connection does not need to be present.

There is a connection for a Broadband Modem which can be automatically switched into use when available. I have checked that this can also be used to link to a normal ADSL, Firewall, Router to add Wifi and/or act as an access point in a different part of a house and/or to add WPA to an older Wifi system.

There is also mention of use of the USB port for a network printer which could be used in that configuration. It is not available in the Firmware supplied and you need to do a simple firmware upgrade. I have done the upgrade twice now without any problems, it involves downloading a single file and running a menu item in the control panel and browsing to the location of your file - it is best to do it with an ethernet cable connection. Overall the menus are slightly easier to follow in the new software. I will report further if and when I try the printer option out.

Cautions: There seems to be a slight leakage of data. There are various timeout functions which require the connection to be checked and also there is a check of the time from a timecode server which can be seen in the log files. The leakage seems to be about 3 Mbytes per hour which means it is advisable to disconnect by unplugging the dongle when the system is not in use. There are various timeouts which can be set to break the connection but the one for a 3G connection seems to be missing in versions 2.08, 2.11 and 2.12 of the software compared to the manual which covers 2.00. That said the first day it was on for 10 hours and the cost was £0.84 with two of us using machines and a total data use of about 50Mbytes according to the monitors on the two machines which accounted for £0.75 on the vodafone tariff we were using. The other feature is that it is set up to automatically disconnect after 8 hours and has to be re-enabled by unplugging the dongle or turning it off and on - in view of the data leak, however small, this is sensible but if you place it in the loft for a good signal it could be inconvenient. Again I will report further when I upgrade the firmware.

Using the Edimax Wireless 3G Broadband Router 3G-6200n as a Print Server

This is largely undocumented and is not available until one has done a Firmware upgrade to 2.12 or higher. This then makes a new menu item available under General settings. I set this up to Enable Print Server, IPR Enable, LPR Enable, Print Server name edimax, and left the Print Name of USB Port as lpt1. That was all the extra activities on the box and in retrospect it would probably have worked without any changes at that end.

I could not find out much even with internet searches so finally I ran Network Tools and did a port scan - that produced a number of pieces of paper out of the printer as well as revealing that there was a LPR printer server was listening on port 515. LPD/LPR is short for line printer daemon/line printer remote, a printer protocol that uses TCP/IP to establish connections between printers and workstations on a network. The LPD software runs as a daemon in the in the print server and the LPR software is already built into most Linux systems. The LPR client sends the print request to the IP address of the LPD printer/server, which in turn queues the file and prints it when the printer becomes available.

I set up the Printer via System -> Administration -> Printers -> Create New Print Queue -> Network -> LPD/LPR Printer and filled in the boxes giving a result as below - note I did not need the Print Server Name at all, just the IP address, port and printer stream.

That got the printer up and running but it could only be accessed by WiFi or the Ethernet connections on the Router as the Router did not pass back to the existing network. I therefore decided to use the Edimax 6200 as the Main Network Router and Firewall and just use my existing Wifi, Firewall, ADSL Router as a fancy modem with a cable connecting it into the WAN connection on the Edimax 6200. An associated advantage is that the Edimax supports WPA as well as WEP which is all I had on my ancient Router. The only problem is that I need to prioritise which Wifi connection is used on the Computers as both are broadcasting still - or turn off Wifi on the old 3Com Router. At least I now do not need to keep a computer running just as a SAMBA print server.

I used the router as a Network Print Server and Wifi Router under firmware version 2.12 for about 6 weeks with no problems until I took it away and tried to use it with for Mobile Broadband. It then became clear that the connection was only staying up for about 1 to two minutes at a time before it disconnected and reconnected. My checks when I had upgraded had been primarily to do with its use as a print server. I downloaded firmware version 2.08 which was still available on the web site and the Mobile Broadband was then perfect again after I had reconfigured all the settings which are lost every time you do a firmware update as you are supposed to do a full restart and load of default settings after every firmware update. There is a facility to save and reload settings but it is still a very undesirable state of affairs and I have contacted the Edimax support and await a response. This may only a problem with my Vodafone Mobile Broadband dongle which is badged as a K3565 but is normally detected as a Hauwei E160E which it is a version of.

14th November 2009

PHP Errors on Website Forms

I suddenly started having errors on the form handlers on the web sites I look after. It turned out that the Hosting Service had changed the level of error reporting and aborted when there were notices such as unset variables coming from blank boxes in a form. I sorted out some of the inputs with tests but also used the lines

// Report all errors except E_NOTICE
// This is the default value set in php.ini
error_reporting(E_ALL ^ E_NOTICE);

17th November 2009

Rebuilding Pauline's Toshiba Satellite Pro L20

I have been going through my procedures for Backing up and Synchronising again. This was provoked by the hard drive failing on Pauline's Toshiba Satellite Pro laptop which is the machine she uses at home most of the time. We lost very little of the Ubuntu system and our data but the Windows system was completely lost. The hard drive was accessible through a panel on the underside and a much larger replacement drive of 120 Gbytes (to match the MSI Winds was obtained and fitted.

The Toshiba was running under Ubuntu Jaunty Jackalope and we had an issue with poor sound which would randomly stop leaving only clicking and also video playback would freeze so this seemed to also be the time to try out the new Karmic Koala. This fixed the sound problem completely but gave another problem with the Wifi drivers if one wanted to use WAP security with the newly developed drivers available in the kernel with the particular Wifi card. Unfortunately the kernel builders had been so confident that they had removed support this time for the proprietary MadWifi driver which had been available in Jaunty. We experimented with fixing the audio versus fixing the Wifi and the upgrade was so much better we chose to go that way and make and install MadWifi drivers for karmic - this is not difficult but will need to be repeated for every kernel update like we used to have to do on the MSI Wind.

Using MadWifi drivers for the Atheros Wifi card under Karmic Koala

This is needed to be able to use the Wifi built into our Toshiba Satellite Pro L20 as the new ath5k driver now built into the latest kernels does not yet support the Atheros driver version used in the Satellite Pro L20 and a number of other machines well if you want to use WEP or WAP security.

The following is my procedure for use under Ubuntu Karmic Koala using Grub 2 which is the default for a new install if you still have the original Grub loader then you need to modify different files to blacklist the ath5k driver.

First install the utilities for building a kernel driver by typing the following in a terminal:

sudo apt-get install build-essential

Now sort out which drivers will be available by typing the following in a terminal:

sudo gedit /etc/modprobe.d/blacklist-ath.conf

change the last line by adding a # at the start to comment out the blacklisting of the MadWifi driver so it reads # blacklist ath_pci and save it then type the following in a terminal:

sudo gedit /etc/modprobe.d/blacklist.conf

now add blacklist ath5k on a new line at the end.

These two actions mean we will be able to use the new MadWifi driver ath_pci instead of the built in driver ath5k.

We now need to obtain the new driver by downloading the latest version from http://snapshots.madwifi-project.org/madwifi-0.9.4-current.tar.gz , it will initially download to the desktop where you can unpack the archive by double clicking it and dragging the folder within it to somewhere you can easily find it as you need to compile it again every time you update kernel - the best place is your home folder and also renaming it to madwifi from a name which includes the version number. The follow steps assume that is done and the folder is in . Once this has been done we need to compile the driver and install it into the kernel. The stages above only need doing once - the following need to be repeated every time the kernel is updated - you will know when as the Wifi will stop working!

The following is a list of the commands one needs to type in a terminal after every kernel change.

cd madwifi
sudo ./scripts/madwifi-unload

make clean
make
sudo make install

sudo depmod -ae
sudo modprobe -r ath5k
sudo modprobe ath_pci

This reports warnings about an option -e which seem to cause no problems - you can try taking the e off the end of the line sudo depmod -a which should solve some of them but I need to test at the next kernel change.

Why change to Ubuntu 9.10 Karmic Koala?

Ubuntu 9.10 karmic Koala is not a LTS (Long Term Support) version so one has to ask if the advantages are sufficient to merit an upgrade when a LTS version is due in 6 months. I have upgraded 3 of our 5 machines each for different reasons. The Toshiba Satellite Pro L20 lost a hard drive so a new install was required anyway and 9.10 had less problems than 9.04 and was better than 8.04. The HP Compaq DX2250 needs the most up to date versions of software as it is used for Video work and the home build was in need of a reinstall as it had been progressively upgraded from 6.06 to 8.04 Hardy Heron and also needed more disk space and repartitioning of the disk drives. The MSI Winds were not upgraded as there were issues with Webcam support and also random Brightness variations. These seem to be a chip problem and the surprise is more that they work perfectly under Jaunty! I will wait for the next LTS version as they are working well at present.

So what are the major changes:

Faster booting

A much upgraded Audio setup using Pulse Audio

Use of the new Grub 2 boot loader (the old one is being depreciated)

Changes in the internal handling of devices as one the existing layers is also being depreciated.

A new interface for adding and removing Applications.

More up to date versions of many Applications.

Better handling of additional Repositories for software not available directly within Ubuntu or to allow you to use more recent versions. I use them for DVDStyler, KdenLive, Medibuntu (DVD etc handlers), Zygrib (Met Forecast model data) and it looks as if Ubuntu are working towards an automatic upgrade route for Repository users.

There are many changes under the hood (hal and pulse audio to name a couple) and some have had unexpected results on some systems - it is an essential development step before the next LTS version but it does mean that you should check out carefully with a LiveCD version before upgrading. If you are happy with what you have then it is prudent to stay with it until you have had a chance to do some comprehensive web searches for peoples experiences with your own machine and also to read the Ubuntu Release notes with great care. That said you will find that the improvements are well worth getting used to a few changes.

How to change to Ubuntu 9.10 Karmic Koala

This has been a good test of my own procedures and I have updated them where possible.

I firstly backed up the home directory on each machine and re-partitioned the drives so I could use a separate /home directory. I have always regarded that as a high risk 'expert' activity but my latest way seems to be much lower risk and have an easy fallback if it does not work. (GUI and Terminal - moderate experience needed)

The next step is to backup everything once more then do a fresh install into the existing root partition - you can also specify and use the existing and separate home partition but make sure you do not reformat it! (GUI)

You need to modify one file to automatically mount any 'data' drives (partitions) which you use and create folders for their mount points. (Terminal)

You now need to reload all the programs which will find all there setup data in your home partition a reuse it. I have a script to do most of that for my standard system set up. (Terminal)

There are a few programs which need to be installed separately and need some system configuration - Truecrypt is my main example. (GUI and Terminal)

You may need to install extra fonts for OpenOffice (Terminal/GUI)

You need to set up your printer (GUI)

You need to set up your network and sharing (GUI)

You need to set up for file synchronisation (unison and ssh) which involves changes to a couple of configuration files (Terminal).

You may want to make changes to the Grub Boot Loader to speed it up (Terminal)

4th December 2009

Changes needed to procedures for customising Grub 2

The legacy Grub basically used only one configuration file which needed to be customised, namely /boot/grub/menu.lst. Grub 2 uses /boot/grub/grub.cfg which is normally not edited as it is automatically generated by /usr/sbin/grub-mkconfig using templates from /etc/grub.d and settings from /etc/default/grub . There is a vast amount of information at https://help.ubuntu.com/community/Grub2 and the following is just enough to get started on configuration. There is also a program which you can install which will do some of this - use synaptic to to search for startupmanager and install it - but it is very basic to that under the legacy Grub.

/etc/default/grub typically contains:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.

The set of configuration files in /etc/grub.d are run in order to set up and build up the Grub menu.

30_os-prober finds and adds all the other operating systems - it can be inhibited by setting its permissions so it is not executable. You can then customise the other operating systems by adding them to 40_custom having had a look in /boot/grub/grub.cfg to see and copy what you want. If you want to inhibit display of the memory test options then make 20_memtest86+ to be non executable.

Restoring Grub2 after, for example, Loading Windows

If you reload Windows then it will overwrite the Grub2 bootloaded. I have only reloaded once to make sure there was a fresh copy of the MBR after a virus attack with a MMR trojan on a Windows machine. The proceedure I used used and will use for reloading Grub2 is based on https://help.ubuntu.com/community/Grub2#Configuring%20GRUB%202

First you must find out the device name/partition of the installed system (sda1, sdb5, etc). This partition is then located and mounted from the LiveCD. The files are then copied from the LiveCD libraries to the proper locations and MBR.

Boot a LiveCD (Ubuntu 9.10 or later for Grub2).

Determine the partition with the Ubuntu installation by typing in a Terminal

sudo fdisk -l

and/or

sudo blkid

The device/drive is designated below by sdX, with X being the device designation. sda is the first device, sdb is the second, etc. In most cases the MBR will be installed to sda, the first drive on their system. The partition is designated by the Y. The first partition is 1, the second is 2. Note the devices and partitions are counted differently. In my case the Linux root file sytem is typically on /dev/sda4 as I have Windows and data filestems below it

Mount the partition containing the Ubuntu installation by:

sudo mount /dev/sdXY /mnt

eg: sudo mount /dev/sda4

Run the grub-install command to reinstall the GRUB 2 files on the mounted partition to the proper location and to the MBR of the designated device.

sudo grub-install --root-directory=/mnt/ /dev/sdX

eg sudo grub-install --root-directory=/mnt/ /dev/sda

Refresh the GRUB 2 menu with

sudo update-grub

Extended procedures for Synchronisation and backup of a Network of machines.

We have been using Unison to synchronise files held in a set of folders between all our machines with success for many months. The main folders are My Documents, My Pictures, My Web Site, Web Sites, My Teaching and Pauline's Documents.

As well as simple files we also need to synchronise/transfer our emails, browser settings and secure information between machines for backup and for when we go away. The Firefox Browser information and the Thunderbird Email, Contact and Task information is held in Profiles. These are folders which have a consistent set of information and synchronising would move files in both directions and destroy the consistency. They have to be copied as a whole between machines - mirrored rather than synchronised.

A further problem is with encrypted file systems produced by truecrypt - these are saved as a single file and the size and date remain unchanged for security and deniability reasons so the only way one can tell they have changed is by a full comparison which is slow. Again we probably want to mirror them. I have added a lot of comments into the template for my synchronisation 'profile' for unison. So I will just provide a copy below.

# Profile to synchronise from desktop triton-ubuntu to laptop satellite-ubuntu
# with username pcurtis on satellite-ubuntu
#
# This example is set up to mirror folders rather than synchronise

#Paths to Mirror - the mirror is returned to being an exact copy of the source folder.

#The paths are defined by regular expressions which ensure that all the child folders - the
#folders underneath the source - are also mirrored
# The definitions are confusing and the 'root' which is being mirrored is the one that follows the -> for example
# forcepartial = Regex Vaults/.* -> ssh://pcurtis@wind-ubuntu//media/DATA
# mirrors the folder media/DATA/Vaults (and all its children) onto the local machine from wind-ubuntu

#Note - we only force these changes between the two machines which have the 'masters' at the time - and ignore updating the others.

# Options

# When fastcheck is set to true, Unison will use the modification time and length of a
# file as a ‘pseudo inode number’ when scanning replicas for updates, instead of reading the full contents of every file. Faster for Windows file systems.
# fastcheck = true
# Note: fastcheck must be false to detect changes in encrypted truecrypt volumes as the file size and modification dates are kept the same.

# When times is set to true, file modification times (but not directory modtimes) are propagated.
times = true

# When owner is set to true, the owner attributes of the files are synchronized.
#owner = true

# When group is set to true, the group attributes of the files are synchronized.
#group = true

# The integer value of this preference is a mask indicating which permission bits should be synchronized.
# In general we do not want to synchronise the permission bits (or owner and group)
perms = 0o0000

The file structure on my data partition which is mounted as DATA is shown below - the names should make most of it self explanatory

Loss of Calendar and Project Data from Lightning under Karmic Koala

If you have profiles for Thunderbird/Lightning which you have set up in Jaunty or earlier you may find that all the calendars seem to have disappeared when you change to Karmic Koala which is very disturbing. This is because Lightning is now integrated into Thunderbird in Ubuntu rather than just being in the profile and this intergration also includes the extension for Google Calendars. These extensions can not be in two places at once and even if you have not installed it in Ubuntu explicitely the framework is still in place and the extension in your profile conflicts with it and the calendars disappear from view - this is a bug in my book but once you realise what is going on there is, at least, a simple workround.

The workround this is relatively simple if you just have a single profile. First you need to close Thunderbird then uninstall lightning-extension using the Synaptic Package Manager (if it has been installed) - I did a full uninstall which removes all the control files. Once you have removed the conflict you now run Thunderbird with each of the problem profiles and Uninstall the Lightning 0.9 extension using the Extension Manager (Tools -> Add-Ons -> Extensions) - do not fear it does not remove the calendar data from the profiles. When this is complete you can reinstall lightning-extension in Ubuntu which brings in calendar-google-provider and calendar-timezones as dependences with it and the calendars will now reappear when you open Thunderbird. If you have multiple users you should clean out each users profiles before reinstalling with Synaptic.

Workround for bug 432598 in Disable Suspend and Hibernate in Karmic Koala

25 November - 9 December

A case study in Virus Removal

I have recently been trying to rescue a seriously virus infested machine which contained valuable business related information which had to be extracted and I also needed to ascertain how much potential damage had been done.

I knew the machine well and knew that care had been taken to keep it protected at all times. The first I heard was an email asking how to check if the virus checking which was working followed shortly afterwards by another saying that the existing virus checker had been replaced by the latest McAfee and a single virus had been detected and quarantined but could I have a look as the machine was now behaving very oddly.

This is a very long story so I will anticipate some of what comes and at this point put forwards what can only be a supposition as much of the evidence was hidden destroyed by the viruses and the removal tools. The bottom line is that there is a significant chance that the machine had been infected for some time and that the existing virus checker, firewall etc had been compromised. One virus source file had a time stamp from 7 months earlier but that could have been deliberately false. The major problems occurred when the latest version of the virus checker was loaded - it seems it was was also quickly neutralised and a fresh load of less stealthy viruses loaded almost as if it was a malicious attack once the earlier stealth had been detected. The best guess is that the payload had been the banking password stealer detected and the problem was hidden by a rare Master Block Record Rootkit which was one of those detected - this works before even the earliest boot-up checks come into play making it almost impossible to detect once it is in place.

By the time I got the machine, which had only run for a few hours after McAfee was loaded it was riddled and not only the new McAfee installation but most of the usual tools were compromised, some problems were immediately obvious, some I found as I progressed:

The only browser Internet Explorer did not even load properly.

Internet access was restricted.

File Options had been removed from all menus so file extensions could not be displayed and hidden and system files were invisible.

Windows Explorer was partially disabled - no direct access but it could be called by secondary routes such as load and save menus.

The registry editor was disabled.

Continuous attempts were made to write to a memory stick (fortunately write protected) .

Loading programs was hit and miss.

Firefox was loaded with difficulty but internet access was blocked.

The mechanism for DOS (pre Windows startup programs) was blocked.

There were no Windows Install disks, only a total disk image replacement program.

The mechanisms for installing additional features in Office such as output converters were blocked.

Additional considerations were:

It was a special Windows XP operating system for Tablet PCs so normal disks were not going to be suitable.

There were no disks for Office 2003 or One Note 2003 and the disk images had been deleted or were never provided which is unusual on a pre-installed system with no built in CD/DVD reader/writer or software.

This looked a fairly hopeless task but I found that I did have access to the command line and Run and msconfig was still alive so I could kill some of the more suspicious start up programs and eventually loaded the Avast 4.8 Virus checker I have found so effective in the past and SpyBot Search and Destroy 1.6 which I have come to depend on for Malware. SpyBot has the useful option of using a file of recent updates rather than updating over the internet which I took advantage of. Avast took out enough in its initial runs to make progress but many of the files and viruses identified by it and SpyBot could not be removed as they were locked or in use and the startup removal options had been blocked.

I therefore loaded a LiveCd version of Ubuntu 9.10, again it was flaky which I now realise was because of the MBR contamination as even the LiveCDs access disks to determine the disk structure and allow for mounting when it is running. I also used a LiveCD of Parted Magic which does not read the disks during set up to delete a number of the files and step forwards a little.

The big step forwards was when, after a lot of internet research, I found that another and less well known command line tool gpedit.msc was still active and this enabled me to get to a 'policy editor ' which is used to enable and inhibit the use of the registry editor and File Options. Such a tool was needed as the use of the registry editor is controlled by the registry which is bad news.

At this point I could insert a USB memory stick and use it to transfer programs. These included ImgBurn to allow me to write and verify a CD/DVD with the all important .pst file from Outlook with all the Emails, Address book, Tasks and Calendar although I could not be totally sure they were virus free.

I am not quite sure what was the critical activity which restored a lot more functionality. Up to now I could download some files but only in short stages and Firefox did not display the expected download windows and installing usually failed. Windows Update had started working and I think my install of Internet Explorer 8 overwrote or reinstalled some system files and restored full Internet Access, program downloading and program installation. This could equally have been the removal of one of the viruses.

I had by now given up most hopes of saving the machine without a complete re-install from a disk image so I now looked to a long term solution and partitioned the drive to give a NTSF Data partition and three partitions for a Linux root (/) ext3 file system, a separate ext3 partition for /home and swap file. I then installed Ubuntu 9.10 Kalmic Koala with the Grub2 bootloader to dual boot Windows and Ubuntu Linux system. I has some difficulties with the install when it came to the inbuilt partition editor which I now believe was the result of the MBR being virus contaminated but once I had got past that the MBR was overwritten as part of the install of the Grub2 bootloader. I reloaded the Grub2 loader and the MBR again after I discovered the trojan.mebroot virus as a precaution.

This enabled me to nail a few more viruses either directly or by deleting files by booting into Ubuntu. I now loaded the free version of PC Tools Spyware Doctor which is very good at finding viruses and spyware and found 11 more which I removed by registry edits and file deletions from Ubuntu - that brought the total up to just over 100 (one hundred) infections which I removed I had removed and operation was almost back to usual. I also found that the wininit file was reloading 4 of them and that had to be deleted.

PC Tools Spyware Doctor then briefly flagged a file being accessed as a trojan as Avast was doing a full scan (but not flagging it as a virus) and that turned out to be the first hint about the MBR Virus Trojan.mebroot being present. To quote F-Secure "This MBR rootkit Trojan.mebroot is very advanced and probably the stealthiest malware we have seen so far. It keeps the amount of system modifications to a minimum and is very challenging to detect from within the infected system." The first detections by the virus firms seems to have been on November 28th with one detection in Mexico and one in the USA, both 5 days after it hit the machine I was working on so no wonder I had a hard job. I located the file and did a search for files with the the same date and found 10 more which I deleted in from the dual booted Ubuntu sytem. The location of these files also revealed that the remote control facilities allowing complete control of the machine were turned on.

Sophos seemed to have also picked it up on Trojan.mebroot and had a free for 30 days Malware and AntiVirus trial so I downloaded both there rootkit Detection and AV packages, registered and installed them. The Rootkit tool detected nothing more but the AV scan found three more viruses (4 files) which were still in the Linux Trash folder from my own search which implies there were a few more they do not yet detect or have knowledge of. The machine is now very slow to startup (lots of AV checks going on??) and log off.

Another example of the Viruses found is Mal/EncPK-LT which is a banking trojan that disables firewall, steals sensitive financial data (credit card numbers, online banking login details), makes screen snapshots, downloads additional components, and provides a hacker with the remote access to the compromised system. It has the typical stealth-mode characteristics common to Rootkits and downloads/requests other files from Internet to keep updated and introduce new threats.
Contains characteristics of an identified security risk. All of the Zbot viruses found have similar characteristics.

A bluescreen problem during shutdown turned out to be unrelated and due to a Sony DVD Video camera driver conflicting with the updated SP3 version of XP.

Actual independent viruses found have been:

McAfee (After Fresh Install)

Win32.Zbot.Yeth

Avast 4.8 (Total 97 moved into virus chest plus another ~10 deleted in Avast and ~10 files deleted under Linux)

During the period the machine was infected it had been kept up to date as far as Microsoft Automatic Updates to Windows XP including Service Pack 3 but not the Office Updates so that was turned on and Office 2003 and One-Note 2003 were updatedand Windows Defender was also loaded.

The final state is, I believe a safe but somewhat damaged machine which needs to be reloaded once all the information is out of date and an audit trail is no longer required. Any bank account passwords and other passwords need to be changed and it is possible that the machine was under remote control at periods. Several of the various pieces of software used will need to be removed or replaced if the nmachine is ever used again under Windows as a business rather than home computer.

In parallel the machine can be safely used under Ubuntu Linux as it now has a dual booted system. The viruses do not affect Linux but I am wondering if it may be worth adding a Windows Virus checker to protect others from forwarded problem files and emails.

The total time taken was circa 62 hours spread over two weeks.

Final thoughts: the task took much longer than I had imagined would be the case and the infections far worse. It is another indication of the problems in using any Windows system however much care you take over security - once one virus is in then you have little hope of recovery other than a lot of time in the hands of an experienced professional. In this case there would have been almost no chance of recovery without running much of the recovery from a LiveCd then dual booted Linux system which raises the question of why one should not run with a refined version of Linux like Ubuntu from the start. I had to do almost everything in by Article "The Road to Freedom - A progressive migration from Windows to Ubuntu for Safety, Security and Savings in Home Computing" just to rescue a system for a while until it is compromised again. There is an old saying "You can not make a silk purse out of a pigs ear"

Ubuntu 9.10 Kalmic Koala Switch on to full desktop and disk activity finished 1:07 and shutdown from empty desktop 0:11 Total Cycle 1:18

Windows XP Professional SP3 Switch on to full desktop and disk activity finished 18:00 and shutdown from empty desktop 2:35 Total Cycle 20:35

This needs a little breakdown and Windows took 3:30 to a visible desktop, 9:40 to a website loaded in Firefox and 12:07 to the Firewall being active - note the gap.

The times are so slow under Windows because the machine was a state of the art tablet computer 4 years ago but with only 512 Mbytes memory which is totally inadequate for XP Pro Service Pack 3 with redundant Virus Protection, Spyware Protection and a full Firewall whilst Ubuntu will run in 256 Mbytes and is reasonably nimble with 512 Mbytes and, of course, needs no virus protection and is designed without open ports.

Thunderbird Address Books, Mailing Lists and SyncKolab

We have been setting up our Xmas mailing lists using the Mailing List facility in Thunderbird. We found there were a number of unexpected features:

Entries in a Mailing List must have an associated email address although this is not true for entries in an Address Book.

The Email address must be unique.

Errors are not reported and it can take a while for the display to be synchronised with the contents of the Mailing List - this can cause great confusion!

Mailing lists are closely associated with the Address Book they are underneath (children to).

If an Address card is drag and dropped from a different Address Book it will also be added with an associated entry in the parent Address Book if one does not exist.

If an Address card is dragged and dropped and an entry with the same email address already exists in the parent then the contents of the parent will be used.

Changes made in the parent or child Address Card are reflect in both.

Deleting an entry from the parent deletes the child's Address Card and vice versa.

Mailing Lists are not synchronised by SyncKolab.

Categories are lost when synchronising using SyncKolab.

There is no sensible way to print labels.

13th December 2009

Virus Checkers for Linux - ClamAV and ClamTk

There are a number of Virus checkers which run under Linux which are for detection of Windows Viruses. A major reason for there existence is to check for viruses on Email Servers running under Linux but most will also run checks on files and folders. It is therefore possible to check a Windows filesystem on a dual boot machine or from a live CD when it is not running. Many of the virus checkers are from the same providers who make them available for Windows and are, of course, proprietor. These include free versions of my two favourites for Windows, Avast and AVG. Instead I am using the open Source Virus checker ClamAV and its Gui interface ClamTk, they have a good reputation and have a huge virus database which is updated rapidly when new viruses and are detected. The next paragraph has a brief description of ClamAV and its associated packages.

Clam AntiVirus is an anti-virus toolkit for Unix. The main purpose of
this software is the integration with mail servers (attachment
scanning). The package provides a flexible and scalable multi-threaded daemon in the clamav-daemon package, a command-line
scanner in the clamav package, and a tool for automatic updating via
the Internet in the clamav-freshclam package. It features built-in support for various archive formats, including Zip, Tar,
Gzip, Bzip2, OLE2, Cabinet, CHM, BinHex, SIS and others. It has built-in support for almost all mail file formats and for popular document formats including Microsoft
Office and Mac Office files, HTML, RTF and PDF. The virus database is normally updated by the clamav-freshclam package which is automatically loaded alongside ClamAV for Internet updating. An update package package can also be created on an Internet-connected computer and run as a .deb. We do not plan to do any on-access checking or automatic checking of incoming/outgoing email at this point in time so we are not running the daemon and have no overheads other than a small amount of hard disk space unless we are running a specific check.

Ubuntu has all the various packages in the repositories so they can be installed by the Synaptic Package Manager or it can be installed along with a simple but adequate GUI for file and folder testing called clamtk from Add/Remove programs as 'Virus Checker' . The standard install also brings in a package to add virus checking as a right click option in the file browser. I installed it and used the Gui (ClamTk) to check a Windows system on a dual boot computer - you first need to mount the partition with the Windows 'C: Drive' which needs administrative privileges and a password. It automatically updates and can give a comprehensive check - it was slow on an entire 'C: Drive' but found the 'test' files I expected. It also had a number of false alarms detected as PUA.Packed.aspack212, mostly in my Irfanview .dll files which I have been using for years so they were obviously false - two more detections needed a little more investigation but again I concluded they were false but will watch them anyway. The false alarms only occurred when checking using the advanced mode for Potentially Unwanted Applications PUAs which seems to use a heuristic method - this is a known anomaly which I am sure will be solved.

We will ClamAV it to check files from students and others where we want to avoid any chance of passing existing problems on to other users.

There is an updated version of the ClamTk GUI which I downloaded as a .deb and installed as it seemed to offer some better features but I would not regard that as essential. Unless you are very knowledgeable keep to the version in the repository.