Converting from HTML 4.01 Transitional to HTML 5

The conversion from 4.01 to 5 involves three major steps. Firstly the code in the headers needs to be changed, secondly there are tags which are no longer supported and thirdly many important attributes are not supported and the effects have to implemented using Cascading Style Sheets (css). The idea is to separate out the formatting and use CSS to control the style and layout of multiple Web pages all at once. It can also be used in individual tags but in a way that a global find and replace rarely works. To show the scope of the problem the <center> tag has been removed and the attribute align=left|right|center again has disappeared. The css methods for center also differ for inline and block tags. Tables also become a nightmare to understand and multiple tricks required to replace hspace, vspace, padding, cellspacing, border, bordercolor, and background colour etc. Once the initial changes had been made the advantages in flexibility become clear when it came to implementing a Responsive Web Design approach to making the site Mobile-Friendly.

Now lets have a look at a real case which exercises some of the more difficult issues. My standard display of pictures uses a table which has 4 pictures per row centered in its cells against a white background and is centered on the page. It starts as a fixed percentage width until window is shrunk to the size of the images. The image img tag has a width and heigth defines so the page renders correctly before they are loaded. The cells and image should not have a border. The image is actually part of a link which is generated on the fly but that is not of the essence here.

This raises how to center blocks and images and define width of images and tables as well as backgrounds and borders. Some of the css is in the general css file and some is in the table itself or the image. In the css linked file or the header we have.

LiveReload

LiveReload monitors for changes in a folder and automatically reloads the browser if any change is detected. Also, when you change a CSS file or image, the browser is updated instantly without even reloading the page.

Installing LiveReload is a simple process in Linux. To install LiveReload one must first instal rubygems, then guard and finally guard-livereload:

This will watch the directories (that we're working on) for the changes in css, js, htm and html files inside them (or in their subdirectories).

Next instal a LiveReload extension for your web-browser. It is this extension that refreshes the page after receiving command from guard-livereload app. Extensions are available for Chrome, Firefox and Safari.

Once you have it installed on your browser, bring up your terminal and browse to the directory where your web files (html, htm, css, js) are located. Then type guard and press enter. You can watch the operation in the terminal

Open up the HTML file on your browser, and press on the LiveReload button (which is located in the main toolbar in Chrome and Safari & in the add-on toolbar in Firefox). The LiveReload button should now be active (red).

Leave the terminal running and make changes to your site and when you save the file, the changes will be reflected immediately on the browser and you can see a record in the terminal. When you have finished type exit

I run with an editor and a browser in half width windows beside each other and use CtrlS frequently to save and view the changes

April 1st 2015

Password Managers - Keypass2 and Keypass2droid

I have been looking for a good poassword manager to replace the use of files containing the passwords which were stored in truecrypt volumes. Truecrypt is very secure but overall it was aninconvenient and inefficient way to work. Some comprehensive research of possibilities led me to Keypass and its 'clones' looking promising and it is open source. To be more specific I have been looking at the Linux version Keepass 2 and Keypass2droid for the android phones. Both are an inprovement on use of an Openoffice/LibreOffice file within Truecrypt and Keypass2droid in particular seems to be well supported by a keen programmer whilst building round the well developed and proven open source code for Keepass 2.

KeePass 2.x has been writen to run on all Windows systems with Microsoft .NET Framework 2.0 or higher (already included in Windows Vista and higher) and other operating systems (Linux, Mac OS X, etc.) with Mono installed. It is designed so it does not need to be installed; is portable. It is a major rewrite of Keypass 1 with a different database file format and many more features. It is based on .NET but can be run using open source .NET runtimes like Mono for other operating systems like Linux and Mac OS X . For most Linux systems, Mono is offered as a package; it's even already included in the GNOME 2.16 and higher desktop.

Importing existing passwords into Keypass2

Importing Firefox saved passwords

Hopefully you will not have a large number of high security passwords saved in Firefox as they are trivial to view unless you have a master password and even then I do not use them for Banking etc.

The easiest way to transfer them is to use a Firefox plugin called Password Exporter 1.2.1 which can be used to export and import your saved Firefox passwords. After installation, you can export and import your passwords in two ways:

I used the .xml format for the export and it worked fine for my passwords but it has been reported that the .csv format may be better for passwords/sites with strange names.

The file should be exported to an encrypted vault like a truecrypt vault or you should be prepared to securely delete it by, for example, installing secure-delete and use in the terminal from the folder with the file to secure delete:

Importing Passwords in LibreOffice Table (via LibreOffice Calc)

This was a challenge but finally worked and transferred 270 narrowboat corinnapasswords. I had a table in LibreOffice which had columns with labels of: Service | Access | ID | Password | Notes

If the notes or other entries have multiple lines this needs to be changed before going any further. LibreOffice does not have an obvious way but the trick is to use find and replace with other options exposed and Regular Expressions ticked. Put a $ in the Search for and a space or other character such as | space in the replace with and all the returns disappear.

Now we can shift to LibreOffice Calc to get everything set up for an export as a .csv file. Highlight the whole table and do an Edit Copy (click in table then Ctrl A and Cntr V) open a new Librecalc file and Paste.

Now we need to delete unwanted columns, reorder the columns and change the headings to: Account | Login Name | Password | Web Site | Comments

Now we can do a File -> Save As using the Text CSV format and save it in an encrypted or be prepared to securely delete it afterwards.

I also used the Edit Filter Settings tick box and then set the box for Quote all Text Cells - this may not be neccessary butcan do no harm.

Finally we can open Keepass 2 and File -> Import and use the Generic CSV Importer

It should not be too difficult to see how to convert your own table or spreadsheets provided you have columns which roughly correspond or where columns can be merged for the initial transfer - you can sort out each entry the first time you need it and then put it into an appropriate group. The only real problem I had was that the csv import would not take files with extra returns even when they were in "quoted strings" hence the replacement with spaces - what I would do next time is to replace with a vertical line | and space to make it easier to understand latter and edit if required.

Keepass 2 PPA for Early Versions of Ubuntu and Mint

I have one machine still running the Mint 13 LTS version which does not have keepass2 in the repositories but there is a PPA by Julian Taylor so I installed keepass 2 by:

Responsive Web Site Design

This section covers the continuing work to make our website into a truly Responsive design and allow it to satisfy the Google Mobile-Friendly tests. What is needed for the site is a fully Responsive design which covers use of mobile devices (Phones and Tablets) as well as Computers which have now overtaken computers in importance for use of the internet. A 'Mobile First' technique where a design for mobile access is progressively enhanced as the accessing devices become more capable is favoured. The bottom line however is that the site MUST pass the Google tools which simulate the Robots they use to evaluate whether the site is mobile friendly and responsive. The tool is at https://www.google.com/webmasters/tools/mobile-friendly and the current site fails on 3 of the 4 criteria and will move down the rankings as a result.

Initial thoughts on approach

The solution for fully responsive pages has to work with the minimum number of changes to existing pages as the combination of pcurtis.com and uniquelynz.com involves over 500 pages. The current pages use scripts which write html to the document, in particular this is used in the popup/lightbox pictures. This is an approach which is going out of favour but saves a lot of extra html code and gives considerable flexibility. Most of the pictures are now in 'blocks' using tables which are up to 4 wide.

The initial approach was to make additions to the tables so extra row breaks can be added to reduce the width to 3 or 2 columns. This is done adaptively writing the extra 'row code' (</tr><tr>). I found this had a advantageous side effect that larger blocks can be used which can respond to the width. The popup/lightbox image code and the table format is written when the page is first loaded or reloaded. A manual reload can be quite fast as almost everything should be held in cache and only a re-render should need to take place.

It is also possible to automatically call a reload if the aspect ratio of a mobile device changes from Landscape to Portrait or even if a significant change in width is detected. Reloads are already used for the changes from popup to lightbox picture displays. This approach is covered below. The adaptive change using additional writing to the document allows also the top menu to change from 6 wide to 3 x 2.

The pages still need hand crafting but the changes are mostly a series of repetative pastes and an edit of the index parameters. The changes in the Navigation Bar at the top and links at the bottom may just be a batch edit.

Orientation and Window Size Changes

This is a significant issue as the page needs to respond to major changes in window size and orientation. It is possible to set up an event handler to reload the page but this has to be used sparsely to avoid the time involved. There are several events which seem appropriate namely resize and orientationchange but the implementation seems to vary between browsers. orientationchange is obviously only appropriate to mobile browsers.

What I have found so far is:

orientationchange works in mobile chrome and the android built in browser but not in mobile firefox.

resize works in Firefox but fires multiple times (discovered using tests with alerts)

The normal way to reload a window, namely location.reload(false), does not work if used after a resize event in Firefox. I have found evidence from some very old conference postings it was inhibited after a resize deliberately because of the risk of infinite loops.

So I initially made a handler for resize events which checked for the size of change:

This does not call a reload of the page unless the change in width (which is all we are interested in) is significant ie could result in the need for a change of the number of columns displayed in adaptive tables. It is deliberately asymetric as a display which is too wide is less of a problem than where entries are clipped.

I abandoned the use of location.reload(false) which should use the maximum amount of cache and next used window.location.href = window.location.href to force a reload - this still seems to use cache.

Latter tests show that window.location.href = window.location.href fails if there is an anchor (#anchor) defined in Firefox and the latest code uses a test for Firefox which uses location.href=location.pathname whilst other browsers use the alternative approach of history.go(0) which preserves anchors

Changes to the popup/lightbox scripts.

These are the scripts which dynamically write the popup and lightbox code on the page. They have been tidied up and the padding adjusted to save width. The only way to get down to 320 wide screens with two column blocks of pictures was to reduce the size of the icons from 160 x 120 to 156 x 117 pixels just for horizontal icons and screens under 356 wide. This 3% reduction is not noticeable in practice. The new code for the hbox script (which dynamically writes the complex calls for lightbox pictures) is:

The only change in the hpop script is to force use of hbox for everything other than computers and netbooks.

Other changes

The magority of the remaing changes are in the Cascading Style Sheet. The sizes of the header text has been changed and likewise the styles for the Navigation Bar to make them higher and easier for touch sensitive mobile use. These automatically get applied to all pages.

A new style has been defined for the navigation links used at top and bottom of the pages to increase the line spacing to make them 'touch' friendly. These have in general to be applied to these navigation links manually.

Changes needed on each page

This assumes that the pages have already been changed to HTML5 and that the changes to limit the width to 960 have already been made!

Add new class to any Navigation Links at top and bottom of page (if used).

Change all Picture Blocks to new format with ResponsivePictureblockSplit(index) Scripts. May be quickest to paste new blocks and drag across each existing hpop and vpop Scripts.

Copy/Paste a complete new 'footer' to the page from a matching page (Home, Pauline, Howto Articles, Uniquely NZ, Small Firms or Search) which has been modified. (Only needed on pages with picture blocks)

Limiting the width of the pages

The first stage of modifying the web site design was to limit the maximum width of the dislay area. The decision was made to limit it to 960 pixels as that was half of a HD screen width of 1920. For aesthetic reasons I wanted to have a matched background outside the display area for each of the 6 areas of the site. The limiting of the width is easily caried out using a css and a few lines of code on each page. In fact the whole site can be changed with a single change in the css for the body and the code changes are only needed to display different backgrounds under the display areas and the rest of the page. The additions on the page are to add a class to the body tag and to enclose the rest of the page in a <div></div> tag with a class like this.

These changes have been already applied to most of the current pages on the web site.

Switching, Hiding and Showing Blocks of text and pictures

This is one of the main techniques used on my pages for responsive design. It is used to change the footers for Mobile versus Computer and for changing the size of Maps and Images.

/*
Function to allow one to switch between two two blocks for Responsive design.
if rlogic is true display the block within a tag with id of showId
otherwise display the block within a tag with id of hideId
this function MUST FOLLOW the two tag enclosed blocks
*/

Bringing it together

Overall Web Site Responsivity Design

Firstly we have to take into account that itis an existing web site and we do not want to go completely back to scratch. This means that the principle of designing Mobile-First is more difficult but we can modify use the basic philosophy to do a mobile design then work back to the existing design. The main principles are to do with navigation and display of images so lets try to specify the techniques used site wide.

A top Navigation Bar for the 6 main areas with backgrounds match the themes used in those areas,

A matching series of navigation links to the main 6 areas at the bottom of the page. This is sometimes identical to the top bar and is sometimes just centered set of 6 links.

A header with title and sub title with a background themed to the area.

A footer themed to the area which contains Copyright and Feedback Links and information on the last update date on the left side and a image linking to the W3C HTML validator on the far right. Pages which have blocks of images also have links to settings for image display - see more below.

All images on 'travel pages' are single images which are left or right justified with the text wrapping them or are in blocks of up to 4 wide and display further details when one hovers over them on a computer.

All travel images are standardised sizes and can be clicked to display a larger image. The original design used popup windows for the larger images and images from digital cameras (and video) can be displayed in two resolutions.

Most travel pages now offer an option of popup windows or an overlay image (lightbox style)

Many pages have a series of navigation links in a single centered and wrapped line at top and bottom.

Some travel pages also have clickable maps.

Some pages have indexes with multiple levels of links implemented as lists.

The original pages were optimised for screens between 640 and 1024 wide which grew to 800 to 1280 wide.

In most cases the maximum page width has already been restricted to 960 pixels to improve readability.

Pages in the Technical Howto series have large numbers of Screen Shots from computers and laptops which are far bigger and have more detailed than can be displayed in total on a mobile device.

A technique of dynamic (on-the-fly) writing of some of the HTML documents using JavaScript is used to simplify the use of popups and overlay images.

It is clear that some of the techniques do not exist on mobiles or machines with a small and/or touch sensitive screen. The obvious restrictions are:

Popups do not open windows but are implemented as a new tab on most mobile operating systems and lightbox style overlays need to be used.

The concept of displaying information when one hovers over a 'link' or 'image' does not exist on a touch sensitive screen.

Where possible existing techniques appropriate for Netbook, Laptop and Desktop machines with touchpad or mouse and large screens will at present be maintained for those machines.

Only overlay (Lightbox) style image expansion should be used for Mobiles and Tablets with touch sensitive screen.

Clickable links on imagemaps should in general be removed on mobiles/tablets - only very small images (under 320 pixels) may retain them, larger images will scale to fit the screen.

A mechanism to detect and respond to orientation changes will be required.

Viewport information provided by mobile devices will be respected.

Other Decisions

The use of dynamic (on-the-fly) writing of the pages can and will be continued and used as one of the techniques used to responding to different screen sizes and orientation changes - this will on occasion require reloading of the page.

There should be automatic mechanism to reload the page but only when absolutely essential.

The page should respond to changes from portrait to landscape as well as width.

The pages will be converted to HTML5 despite the major work required to convert from HTML 4.01 Transitional

As much as possible of the layout will be determined in the main Cascading Style Sheet and in the Javascript include file.

Wher hand crafting is required breakpoints etc will be found using function calls such as isNetbook() , isComputer() and isTwoColumn().

Cross browser compatibility will a major consideration and the minimum 'standard' of javascript(1.2) and css (2.0) will be used.

All pages will have a link to the HTML validator and be checked against it for HTML and css.

Selected pages will be periodically checked against the Google 'Mobile Friendly' test page.

Breakpoints

There will need to be a number of 'breakpoints' where the layout changes but the number should be minimised. The main width breakpoints are determined by:

The Navigation Bar which needs to split from 6 x 1 to 3 x 2

The widely used blocks of images will need to reduced from 4 columns to 3 then 2 which can just cope with two 160 pixel wide images in the minimum Viewport size which one needs to design for.

The major transition from a mobile/tablet to a netbook/laptop/desktop

Although not optimum the same breakpoint in width can be used for:

3 - 4 columns in picture blocks

Breaking the Navigation Bar into two rows

The transition from mobile/tablet to netbook/laptop/desktop in terms of width although other factors are involved.

The end result is that thereonly need to be two breakpoints which cause a potential reload:

Minimum design width 320 pixels ie before scolling required

With some additional 'shrinkage' below 360 in padding and minor image dimensions reduction not involving a reload.

Break from 2 columns to 3 columns ~550 pixels - also used for image maps and some screen shots.

Break from 3 to 4 columns also used for Navigation Bar at ~720 pixels - also used for some larger screen shots.

Maximum width is limited to 960 pixels.

Image Maps

Image maps can not be scaled so there has to be a choice based on their size as to whether just an image is displayed which scales or the full map which is a fixed size based on the image size. A good criteria is probably to use a break-test of two column to change from a full display of the picture map to a simple image with scaling allowed. In practice this should enable image maps on most mobiles in landscape view. This will need a script which allows swopping between the two display options..

Other fixed images - screendumps

I looked at various options including changing to a scalable width with potential jitter or use a scaled down icon (160px) and lightbox overlay (without using a script). In the end I decided that making sure that the pages should work just as before on a laptop or desktop should take priority. In any case the screen dumps are from desktop or laptop and those who need them will be using such a machine when they need them. I therefore ended up with a similar solution to that developed for image maps. Depending on the image size I choose to display either

The original image with width and height specified so that the page can be rendered correctly before they are loaded.

A scalable image

The breakpoint is chosen using the nearest 'column' breakpoint.

An example (which also showies the original image commented out) and making use of my ResponsiveShowHide function is:

/*
Function to allow one to switch between two two blocks for Responsive design.
if rlogic is true display the block within a tag with id of showId
otherwise display the block within a tag with id of hideId
this function MUST FOLLOW the two tag enclosed blocks
*/

Handling Settings and the Footer

The is the last area where we need to look at responsive design. Currently the footer is where the settings for display of images are changed. Again a lot of options were tried out but the best seemed to be to switch betweenthe existing footer on wide displays with a navigation bar and two settings 'links' to a compact display without settings or navigation bar and instead have a popup menu with combined navigation and settings. This did not require any extra JavaScript - only some more CSS. It looks quite complex as the settings part is writen on the fly and changes in response to screen width and whether it is a computer or netbook.

Creating a Menu

The following shows the important section where the menu is created and switched and toggled on and off by an onClick on the menu image.

Lists

We have now covered most of the changes to make the pages responsive. There are however some more changes which will make them display better on a small screen. Firstly the indentation on lists needs to be reduced, particularly for those which are deeply indented and secondly the spacing needs to be increased if they are used as a menu or index with links to increase the spacing for touch screens. These changes are made by defining some new classes in the CSS for the various list types I use and by padding the list items. The list padding needs to only be applied when required but the reduction in indentation could possibly be global in the future.

Navigation Bars

The same requirement to increase the spacing between links applies to Navigation Bars but here the solution is even simpler and just involves increasing the line spacing. The bar is also centered at the same time.

p.navigationbar {
text-align:center;
line-height:1.6;
}

27th June 2015

DavMail Gateway to access a Microsoft Exchange Server from Linux

Introduction to the Microsoft Exchange Server

Many large firms, government agencies and universities use Microsoft Exchange. Microsoft Exchange is a proprietor collaborative application product developed by Microsoft. Exchange's major features consist of electronic mail, calendaring, contacts and tasks; support for mobile and web-based access to information; and support for data storage. It uses a non standard proprietory protocol called EMAPI owned and licensed by Microsoft.

Client Software to Access the Exchange Server

Microsoft Outlook is a personal information manager, available separately application and as a part of the various Microsoft Office suites. It is the optimum client designed to work with Microsoft Exchange Server and it is claimed to have 500 million users. As well as e-mail, it also includes a calendar, task manager, contact manager, note taking, a journal and web browsing. It can be used as a stand-alone application, or can work with Microsoft Exchange Server and Microsoft SharePoint Server for multiple users in an organization, such as shared mailboxes and calendars, Exchange public folders, SharePoint lists and meeting schedules. When this article was first writen the current version was Microsoft Outlook 2010 for Windows and 2011 for Mac and they 'only' cost£119.99 as a standalone program. Not surprisingly there is no version for Linux.

The Outlook Web App (OWA), originally called Outlook Web Access, is a webmail service of Microsoft Exchange Server. The web interface of OWA resembles the interface in Microsoft Outlook. OWA is used to access e-mail, calendars, contacts, tasks, and other mailbox content via a web browser and offers much of the functionality of Microsoft Outlook. The most important difference is that Outlook allows users to work when an internet connection is unavailable, whereas OWA requires a fast internet connection to function. The OWA interface comes in two flavours, one with a complete feature set and an alternative 'Lite' version with reduced functionality. The full version requires Internet Explorer 7 or later, Mozilla Firefox 3.01 and later, Google Chrome or Apple Safari 3.1 whilst the Lite version is rendered in most other browsers.

Mobile Devices Most Mobile Devices, Android and iPhone certainly, have a built in Exchange Client usual called Exchange Active Sync which offers subset of the full Outlook Exchange Client. Although Exchange Active Sync is available on Android which uses the Linux kernel, it does not seem to be available for Linux.

Access via IMAP and POP. The Outlook Exchange Server email can be set up to be accessible via IMAP or POP and SMTP although, not surprisingly, Microsoft has not implemented the full standards and conventions used by almost everyone else. These is the best way to be able to continue to work when an internet connection is unavailable or expensive and you can not afford to purchase Outlook or one uses Linux. It can also be used on a mobile. POP and IMAP is not switched on by default and many firms do not implement it.

DavMail Gateway. This is the subject of this paper and allows standard email clients on to have access through the OWA interface protocols. This is available for Linux and every issue is checked under Ubuntu.

The DavMail Gateway

None of the solutions above provide a fully satisfactory solution if one is using Linux, even if IMAP and POP are enabled. One either needs to be online or one sacrifices the possibility of Calenders and Contacts. This is where the DavMail Gateway comes into its own.

DavMail Gateway is an open-source exchange gateway that enables users to use any mail/calendar client (e.g. Thunderbird with Lightning or Apple iCal) with any Microsoft Exchange server from most operating systems. DavMail gateway is implemented in Java and should run on any platform. DavMail is written in Java andreleases are officially tested on Windows, Ubuntu Linux and Mac OS X platforms. DavMail Gateway's operation is simple yet elegant: it uses Outlook Web Access to retrieve your e-mail, calendar and contacts from your employer's Exchange server and then retransmit them to your local client using standard compliant protocols. This means LDAP for global address book, SMTP to send messages, IMAP to browse messages on the server in any folder, POP to retrieve inbox messages only, Caldav for calendar support and Carddav for personal contacts sync. Thus any standard compliant client can be used with Microsoft Exchange. In the case of Linux the obvious candidate is Thunderbird with the Lightning extension for Calenders and Contacts.

The DavMail Gateway is a small program which has to be running whilst Exchange is being accessed. On Linux Mint it sits in the System tray and the settings etc can be accessed on a right click menu. It is run from the normal menu and can be added to the start up menu so that it is always running whilst the machine is on - I have not done that until I know the resources required, at a first look it seems that there is no CPU load but a memory footprint from Java of 150 Mbytes which seems very high.

Installing DavMail

When this section was last updated in June 2015 Linux Mint was at version 17.1 which was based on Ubuntu 14.04.

DavMail is not available in the Ubuntu/Mint Repositories and I have not found a PPA yet which has it so it has to be downloaded as a .deb file from the website - mine was davmail_4.6.1-2343-1_all.deb (5 mbytes) and I downloaded to the desktop. I used the gdebi installer which should be called if you double click on the package or via the right click menu. gdebi automatically installs any dependencies (other packages which are required for it to run). On my system two small extra packages were installed (total download size 200 kbytes). I am not sure if gdebi is installed by default in Ubuntu or Mint so you may need to install it using the synaptic package manager or from the terminal.

sudo apt-get install gdebi

DavMail is a Java program so Java will be installed if you do not already have it - that is a big download so do it on Wifi not mobile! You should already have the Open Source Java (openjdk-7) installed and that should satisfied the dependencies without the real Oracle Java. I have Oracle Java 8 installed on my own machine as well and that machine is configured to use it so I can not check for problems with the Open Source Java until I have checked on another machine. Installing Oracle Java is not to be undertaken lightly and involves 60 Mbyte downloads!

Once I had run gdebi that was it - it was working 'out of the box' when I ran it from the main menu sitting in the system tray. Only one parameter has to be setup in DavMail Settings which is the OWA (Exchange) URL. Everything else should be left at the default settings unless you have a port conflict - the screen below makes it clear.

Thunderbird Settings when using the Davmail Gateway

Thunderbird is set up just as usual except that the servers are all localhost and the ports are as per those set in the Davmail Gateway ie POP is 1110, IMAP is 1143 and SMTP is 1025. At present we have not tried calendering etc. You get to Thunderbird Accounts by Edit -> Account Settings -> Account Actions -> Add Mail Account and use Manual Setup. The POP and SMTP settings should look like thiswhen you are finished:

And thats it. Run DavMail first, then Thunderbird and collect and send your mail! DavMail can be closed by a right click -> Exit on the icon in the system tray.

Compatibility of DavMail Gateway with existing Exchange POP and IMAP accounts in Thunderbird.

My tests so far have involved making new POP and IMAP accounts when using DavMail. If the DavMail Gateway is completely compatible IMAP and POP accessed directly from the server it should be possible to just change the Server Name, Port and Authentication on an existing account. This would avoid having to reload all the emails and maintain an existing local filing system.

My initial tests on a POP mailbox show that all the existing folders remain as expected but after the password is input all the emails on the server start to download, oldest first. This will give duplicates in the inbox of emails which have not been files or deleted. Next time I will move all the existing inbox messages to a temporary folder until the whole inbox is downloaded at which point the older emails can be deleted and the temporary folder moved back.

I have only partially tested the compatibility with existing accounts and strongly suggest that the profile is backed up before any such tests, especially if emails are partially downloaded as a limit on size has been set when using POP.

Conclusions

I have only been able to carry out tests on one version of Exchange Server so far and have only used new Thunderbird Accounts. I have only tested email not Calendars or Contacts. Within those boundaries it has all worked exactly as expected out-of-the-box on a mailbox with several thousand emails.

My testing has only been on Linux but this should allow use of Thunderbird/Lightning on Windows systems as free Open Source alternative to Outlook.

18th August 2015

Rework of Small Firms Pages

There have recently been many changes that effect the use of the Internet by Small Firms

22 September 2015

Audit of Web Site and Progressive Updates

It is coming up to 20 years since the web site was started and the structure and state of updates is complex.

In no initial order and priority some of the features and challenges are:

The web site actually comprises two domains and 5 areas. The domains are intended to only be linked in the entry pages to each area by having absolute addressing when uploaded but relative addressing during development and to provide an offline version.

There are over 30,000 files including images. This was discovered because the new editor Brackets that I am looking at has a limit of 30,000 files in active use and I had to use an extension which enables me to exclude some folders and types of files

There are at least two .css files (general.css and responsive.css) and several .js files depending on the status of pages in use in the site:

general.css (283)

responsive.css (381)

popup.js (113)

epopup.js (64)

ebox.js (68)

rbox.js (352)

rbox2.js (26)

Changes to files

In getting Brackets to work I had to make a number of changes to the project files:

Many of the backup and test files were moved to a new folder called archive. In particular I wanted to cut down the number of .css and .js files in sight of Brackets.

A number of the legacy files had to be converted to UTF-8 Unicode encoding using Cntr J in Dreamweaver to access the encoding tab. Brackets will only open UTF-8 encoded files and all the HTML5 files have to be UTF-8 anyway.

Backward Compatibility

popup.js (113) to epopup.js (64)

popup.js offers popitup5() as the top level function with tidy() and early versions of browser_caution()

epopup.js keeps backward compatibility with:

browser_caution()

popitup5()

fbpopitup()

tidy()

The main additions between popup.js and epopup.js are:

Routines to simulate functions used on older pages. popitup(), ppopitup, vpopitup(0 and pvpopitup()

General functions for cookies initially used to remember if user has broadband and a large screen.

Functions called for vertical and horizontal popups of two different sizes (image files end in i.jpg for icon, w.jpg for small image (400*300) and b.jpg for large images (600x450):

popitup2h()

popitup2v()

Functions to put in the HTML to provide simplified calls for vertical and horizontal popups with a doubleClick event handler calling toggle() so that large popups can be displayed if user has broadband.

hpop()

vpop()

Code potentially used to calculate the connection speed by measuring the time to load one of the images on the site. Never used on any pages that I can find other than a test file bandwidth_test.htm which has no links to it.

Code to hide and expand blocks

function ExpandOn(topicID)

function ExpandOff(topicID)

function ExpandToggle(topicID)

Changes epopup.js to rbox.js which affect backward compatibility.

Code to determine download speeds dropped - never used

Major changes to hpop() and vpop() to provide a choice of existing popup code or code to provide an lightbox style overlay. Calls are still identical but default of overlay may apply with no mechanism to change on page, only using settings page.

BIG PROBLEM Lightbox Overlays require jquery and lightbox to be available which means three extra lines of includes are required in the head section and overlays prefer HTML5

All code changed to use styles rather than attributes. Should have no impact

Extra cookie to save preference for overlay or popup. See above and the extra cookie forces one towards an EU cookie warning banner.

More sophisticated determination of screen sizes etc and forcing of use of overlay on mobiles. Again a positive benefit.

Event handlers are added to the page to detect clicks in the body and to inject code for the Cookie banner after the page has completed loading. The click detection should not be loaded unless there is a specific class added to a tag somewhere on the page so there should be no negative impacts although some tests are desirable.

Tidy of files

A number of the older files were backups or drafts and these were moved to the archive folder which reduced the number of files with the ebox.js javascript include file and the remaining 12 were individually checked and the javascript file changed to rbox.js.

rbox.js had already been updated to be the same as rbox2.js so the files with rbox2.js were changed to use rbox.js so the end result is:

general.css (224)

responsive.css (371)

popup.js (116) -> 134 Why did this increase??

epopup.js (69) -> 0

rbox.js (380) -> 436

dbox.js (0) New replacement for rbox2.js for development testing

This implies there are a number of files without a JavaScript include.

I then had an individual look at the files still using epop.js and found many more were draft files which I archived and then changed the remaining with lightbox includes when required because they had popups. This resulted in the final figures.

Need to check that the defaults are being correctly applied.

Checks showed few differences between general.css and responsive.css and general.css updated to responsive.css.

I believe all the files using popup.js are all HTML4.01 and do not use cookies. They are mostly under a 'legacy' heading.

Brackets

Brackets is a OpenSource code editor with a number of very useful features and a quite a few irritations.

Features

Inline Editors: Instead of jumping between file tabs, Brackets lets you open a window into the code. When you want to work on the CSS that applies to a specific ID one puts one's mouse cursor on that ID, push Ctrl+E and Brackets will show you all the CSS selectors with that ID in an inline window so you can work on your code side-by-side without any popups. The same works with JavaScript if the cusor is on a function call

Live Preview: Provides a real-time connection to a Chrome browser. Make changes to CSS and HTML and you'll instantly see those changes on screen. Also see where your CSS selector is being applied in the browser by simply putting your cursor on it. It's the power of a code editor with the convenience of in-browser devopment tools. You have to refresh after makin JavaScript changes in the Inline Editor.

Code Hinting: Evoked as one types or by Ctrl Space

Instant Search in Files: Searching across files is very fast and results are now displayed incrementally and updated as you type. In the case of Search and Replace you can select which occurances to change.

Very extendable: Large number of extensions in existance with built in mechanism for instalation.

Open Source and therefore free

Irritations and some workrounds via Extensions

Inline Editor lacks inteligence in determining priotity in displaying list of selections for the Inline Editor.

Spell checker not built in and only extension is to an online version and its use disabled code hinting.

No UTF-8 support and will not even display non UTF files. I discover many of my legacy files from Windoz days were not UTF encoded and I had to convert in Dreamweaver or Gedit. No extension to allow instant conversion yet available.

Not a tabbed approach but this allows it to support split screens - Overall the split screen means the approach has more Pros than Cons. There is an extension which adds tabs but it actually offers no advantages and confirmed to me that the approach adopted may be correct.

Pages do not remain easily accessible after access, only after a change. The only way to add a page in the 'working files' list is to make a change after which it stays in the list even if it is undone.

The Right Click Menu lacks basic functionality including Cut and Past - needs an extension to give partial support. I am using extensions such as:

htmlFormat is a further extension which adds even more called which adds a drop down meny to the top toolbar and right click menu and also the bold, underline and italic etc formating normally available via Cntr B Cntl U and Cntr I

Drag and drop Disabled you go into prefernces and find it and change in your over-ride file .brackets.json in the project folder.

Limit of 30,000 files handled by the system - sounds a lot but I have over that because of all the image files. Can be got round by various extensions such as:

brackets-file-tree-exclude by Martin Zagora which excludes folders from the Brackets file system to avoid the 30,000 file limit. The excluded files and folders need to be setup in .brackets.json in ones projects folder.

Many Extensions are flaky, maybe just under Linux but I suspect there is no quality control. I have discoved there is an extension called:

Extensions Rating which extends the extension manager so that it has extra information from GIT such as the number of downloads, trending, stars, when updated last etc which gives an indication of what might be OK.

Other Extensions I use are:

Brackets Icons is an extension which adds File Icons into the Brackets file tree

Brackets Git integrates Git into Brackets

Conclusions on Brackets

I have found it is a useful addition to my toolbox and I suspect my use will grow with time especially when the bugs and rough edges are ironed out.

I find the site wide search to be very good and the Preview is useful for tuning some appearance by editing styles and classes. It is not a WISIWIG editor making text entry less good so I am now using a mixture of my early Dreamweaver MX running under Wine and Brackets. Dreamweaver does not support HTML5 specifically which Brackets does and does not run JavaScript in its WISIWIG Mode. Both can be open to the same file and both reload if the file changes in the other editor so as long as you do not have unsavd changes in both it works well.

I have a Sumvision SV-05B Bluetooth keyboard which I bought to use with our Android Pads and Phones which is basically an American keyboard layout but with a bias towards an Apple Command Key Labels.

Some of the standard codes do not appear or are interchanged. I have been looking for workarounds and got started with http://www.windowsbbs.com/pc-hardware/31308-vertical-line-uk-keyboard.html which gave some examples of the use of AltGr (The right hand Graphics Key) and led to further experimentation. The following work on my Sumvision SV-05B Bluetooth keyboard ffom 7dayshop with Linux.

These may vary between keyboards but it gives a start for experimentation. AltGr means "Alt Graphics" and it is the international equivalent of the right-side Alt key. Hold it down to type accented vowels, remapped punctuation (`~#[]{}\|) and other characters. Also see https://en.wikipedia.org/wiki/AltGr_key for lots of information on keyboard layouts and AltGr under different operating systems.

In summary, you may have to experiment with your keyboard and operating system if you can not change the keyboard mapping to exactly match your keyboard

Patching a kernel

This article has been written in response to a requirement to patch a new xhci (USB) driver into the kernel for my Helios Ultrabook if only to make sure that the problem was understood. It is therefore not comprehensive and adopts simple solutions to a particular problem rather than an approach which is very flexible. I am not looking to patching the mainline kernels at kernel.org but those which have been adapted and developed for Debian/Ubuntu and thus for Linux Mint. In general these have a longer life and have more uppstream changes backported into them than the original kernels they are based on but arguably this may lead to less continuity between kernels with changes backported to sat wily and trusty but not to vivid. It also leads to some confusion as to patch say the 3.19 kernel used by Mint 17.3 you need to work on ubuntu-vivid and for 4.2 one works on ubuntu-wily.

Installing required software

Set up build environment

If you've not built a kernel on your system before, there are some packages needed and dependencies which need to be satisfied before you can successfully build a kernel. The article I quote just says you can get these installed with:

sudo apt-get build-dep linux-image-$(uname -r)

but this only works if you have first enabled the source code repositories in Ubuntu and Mint. Open Software Sources from the Menu and under Official repositories tick the box under Source Code to enable source code repositories then click Update Cache before the terminal string above.

Install other required packages

sudo apt-get install linux-cloud-tools-common linux-tools-common

These may be needed when you install your compiled kernel later.

Finding and downloading the Kernel Source using git.

All of the Ubuntu Kernel source is maintained under git. The source for each release is maintained in its own git repository on kernel.ubuntu.com. To obtain a local copy you can simply git clone the repository for the release you are interested in as shown below. For example to obtain the vivid tree which matches Mint 17.3:

Once you have cloned the source from git you can see the individual versions (3.19.0-39 etc) has a tag. You can see the huge list of tags by changing into the folder and doing git tag as below but it is better to use grep to filter out the ones of interest:

cd ubuntu-vivid
git tag
| grep -i 3.19.0-3

and you can then get to any earlier kernel version to see or work on by

git checkout tagname

If you want to patch at that point it is best to create a branch even if you delete it afterwards for example if you want to patch the base version of Mint 17.3 which uses kernel 3.19.0-32.37 you would create a branch for the work like this.

git branch -b working_3.19.0-32.37 Ubuntu-3.19.0-32.37
and after you have finished you have the option of deleting it by

git branch -d working_3.19.0-32.37

If you plan to use git extensively or send any patches back git itself should have a little setup done.

Applying the Patch

Firstly you need to know a few things about the patch programme.

You call it like:

patch -pnum < patchfile.patch

This tells us that the patchfile contains the information to on the files it will be patching, and often there are several. However the folder structure can be different so there is an option to work from up and down a folder tree with -pnum where num is the number of levels to strip off from each filename found in the patch file. This is often -p1 because of the way the patch is created from kernels in separate folders.

There are lots of options but two I find useful are --ignore-whitespace (matches patterns loosely in case tabs and spaces have been munged in the file by email or browsers) and --dry-run (for testing you have the -p value correct)

In this case the patch is in a file at http://linux-kernel.2935.n7.nabble.com/TESTPATCH-v2-xhci-fix-usb2-resume-timing-and-races-td1250796.html . One does not need to be too careful about where to cut out the patch as the patch program is good at only using the important bits but I cut out after the diff -- git line and before the 1.9.1 and put it into a new file xhci.patch - Remember this is NOT the final patch but a test so it is for informatio or experiment on a backed up system! The patch was intended for kernel 4.3 and the second file xhci-ring.c was slightly shorter in 4.19 but the matching in patch found the right place and reported the difference in lines of 9. I did need the --ignore-whitespace and I did a dry run first which looked like the following when it was run from the ubuntu-vivid folder.

patch -p1 --ignore-whitespace --dry-run < xhci.patch

and then removed the --dry-run. As this was my first try I did inspect the files very carefully to see that it looked as if it had worked properly.

Building the kernel

Building the kernel is actually quite easy but is slow - even on my fairly fast i7 laptop it takes nearly an hour. make sure your your working folder is the the root of the kernel source tree and then type the following commands:

I am not sure if the chmod are neccesary but several howto pages include them

fakeroot is an interesting command that wraps some other commands so it looks as if they are being run by root without needing sudo and allows the building of packages. If the build is successful, a set of at least 4 .deb binary package files will be produced in the folder above the build root folder. These will have names matching the 'tag' you have chosen and likewise when you install. This means you will be overwriting the existing kernel or risking having an update overwrite it. In order to make your kernel "newer" than the stock Ubuntu kernel from which you are based you should change the version number. The version number is in the first line of the debian.master/changelog and I incremented the very last part of the number ie in the example from 3.19.0-32.37 to 3.19.0-32.38 or 3.19.0-39.44 to 3.19.0-39.45

gedit debian.master/changelog

This will help identify your kernel when running as it also appears in uname -a. Note also that when a new Ubuntu kernel is released that will be newer than your kernel (which needs regenerating), so care is needed when upgrading. Mint does not automatically upgrade kernels which is an advantage.

Testing the New Kernel

Install the package set on your build machine by something like

sudo dpkg -i linux*3.19.0-32.37.38*.deb
sudo reboot

I found that I had 6 files and the ones for tools had dependency problems and are not required so it may be best to install explicitely the 4 one needs. I may try to find out about the missing dependencies as the linux-tools.... is useful for some diagnostics such as perf.

During rebooting you may need to select the new kernel. If it does not work properly you old kernel should still be available if you have made sure you used a different name or had another stock kernel loaded.

Maintaining your changes, and Updating with the latest Ubuntu Kernels

This is where the approach using git comes into its own as one does not have to download the whole kernel tree every time it is updated. I will not go much further here as this was only intended to be a one off test when I started. To go further we need to discuss staging, commits, pull and the use of .gitignore

Updating to latest version of git kernels can be done without downloading the whole package again by

git pull

but you will probably need to commit any changes on your patched branches first

Before You Leave

I would be very pleased if visitors could spare a little time to give us some feedback - it is the only way we know who has visited the site, if it is useful and how we should develop it's content and the techniques used. I would be delighted if you could send comments or just let me know you have visited by sending a quick Message.