The Essentials of Data Science: Knowledge Discovery Using R presents the concepts of data science through a hands-on approach using free and open source software. It systematically drives an accessible journey through data analysis and machine learning to discover and share knowledge from data.

Building on over thirty years’ experience in teaching and practising data science, the author encourages a programming-by-example approach to ensure students and practitioners attune to the practise of data science while building their data skills. Proven frameworks are provided as reusable templates. Real world case studies then provide insight for the data scientist to swiftly adapt the templates to new tasks and datasets.

The book begins by introducing data science. It then reviews R’s capabilities for analysing data by writing computer programs. These programs are developed and explained step by step. From analysing and visualising data, the framework moves on to tried and tested machine learning techniques for predictive modelling and knowledge discovery. Literate programming and a consistent style are a focus throughout the book.

]]>Open Source R on the Azure Ubuntu Data Science Virtual Machinehttps://togaware.com/the-azure-linux-data-science-virtual-machine/
Sun, 06 Aug 2017 16:07:29 +0000http://togaware.com/?p=497Data scientists rely on the freedom to innovate that is afforded by open source software. We often deploy an open source software stack based on Ubuntu GNU/Linux and the R Statistical Software. This provides a powerful environment for the management, wrangling, analysis, modeling, and presentation of data within a tool that supports machine learning and artificial intelligence, including deep neural networks in R.

Whilst the open source software stack is also usually free of licensing fees we do still need to buy hardware on which to carry out our data science activities. Our own desktop and laptop computers will often suffice but as more data becomes available and our algorithms become more complex, having access to a Data Science Super Computer could be handy. The cloud offers cheap access to compute when you need it and the Azure Ubuntu Data Science Virtual Machine (DSVM) has become a great platform for my data science when I need it. The Ubuntu DSVM comes pre-installed with an extensive suite of all of the open source software that I need as a data scientist (including Rattle and RStudio).

A new data science virtual machine can be deployed with with a few clicks and some minimal information in less than 5 minutes. As our data and compute needs grow it can be resized to suit. Paying for just the compute as required (e.g., at 25 cents per hour) is an attractive proposition and powering down the server when not in use saves me considerably compared to having a departmental server running full time, irrespective of its workload. When not required we can deallocate the server to cost us nothing. There is no need for expensive high-specification hardware sitting on-premise waiting for the high demand loads when they are needed. Simply allocate and resize the virtual machine as and when needed and pay for the hardware you need when you need, not just in case you need it.

The version of R provided with the Linux Data Science Virtual Machine is Microsoft’s R Server (closed source). This is based on the open source version of R but with added support for beyond RAM datasets of any size with parallel implementations of many of the machine learning algorithms for the data scientist. In the instructions below though please note that we replace Microsoft R Server as the default R with open source R. Both are then concurrently available on the server.

I begin with a link to obtaining a free trial subscription (if you don’t already have an Azure subscription) and then continue to set up the Ubuntu DSVM using the Azure Portal and configuring the new server with various extra Linux packages (that are not yet on the DSVM by default – but stay tuned) as well as an updated version of open source R and Rattle. Note that the deployment and setup of the DSVM can also be completed from R running on our own laptops or desktops using our new AzureDSVM R package. This then allows the process to be programmed.

The following looks like a lot of steps, and maybe so, but each is simple and the whole process is really straight forward. If you disagree, please let me know and we’ll work on it.

Obtain an Azure subscription

A free trial subscription is available from azure.com. This is useful to get a feel for the capabilities of the Azure cloud and the costs involved. Costs apply only for the time the DSVM is deployed (irrespective of how much the CPU is utilised when it is deployed) so it is good practise to stop the server if you don’t need it for a period of time.

Select the Data Science Virtual Machine for Linux (Ubuntu) from the search results.

Read the description to see if it matches your requirements and then click on Create.

Setup the Basics

Name the machine. E.g., dsvmxyz01.

Keep SSD as the VM disk type.

Provide a Username and Password. E.g., xyz and h%nHs72Gs#jK. (Using an SSH public key is preferred but beyond the scope of this introduction and can be set up later.)

Choose your Subscription.

Create a new Resource group and give it a name. E.g., dsvm_xyz_sea_res. A resource group is a logical collection of resources.

Choose a Location. E.g., Southeast Asia.

Click on OK. Your selections will be validated.

Choose a server Size.

Choose a VM size. The configuration and monthly cost will be displayed for each. I generally start with the cheapest and rescale later as needed. Note that $100 for a month is, very roughly, 15c per hour whilst it is Running and no charge whilst it is Stopped. You can later resize the server if you need a bigger one to get things done more quickly.

Click View all to see all server options.

Once you have decided then click on Select.

In Settings

Check the default information and generally we go with the defaults unless we know otherwise.

Click on OK.

In Purchase

Check the Offer Details, the Summary and the Terms of use.

If all is okay then click on Purchase.

Wait while Deploying Linux Data Science Virtual Machine

This takes about 5 minutes.

The new VM appears in the default Dashboard.

Set up a DNS name label (should be done during set up – how?)

Click the Public IP address

Click Configuration

Provide a DNS name label. E.g., dsvmxyz01.

Click on the Save icon at the top of the tile.

We can now refer to the server as dsvmxyz01.southeastasia.cloudapp.azure.com

You will be asked for the user’s password in order to authorise the running of the RStudio server.

Connect to http://dsvmxyz01.southeastasia.cloudapp.azure.com:8787 You will be warned that the connection is not secure. You should see the RStudion login page and if you are comfortable with the scurity warning then continue to provide your username and password. Note that encrypted RSA is used in transmitting the credentials so I believe it should be secure.

Username: xyz
Password: h%nHs72Gs#jK

Install support packages and latest R

Connect to the DSVM through X2Go for a desktop experience and open up a Terminal Emulator.

Update the operating system, install some utilities, and then reboot the server (note that you do not have to accept the EULA for the msodbcsql package as it is not required for the open source stack and can be removed):

RStudio can be used through the browser from your local machine as we saw above, or else on the remote server’s desktop. For the latter:

Start-up RStudio (click on icon)

Notice message warning about Untrusted application launcher

Click Mark Executable

RStudio will start up.

Graham @ Microsoft

]]>Running an R Workshop on Azure with the Ubuntu Data Science Virtual Machinehttps://togaware.com/running-an-r-workshop-on-azure-linux-data-science-virtual-machine/
Sun, 28 May 2017 02:37:46 +0000http://togaware.com/?p=515The fully open source software stack of the Ubuntu Data Science Virtual Machine (DSVM) hosted on Azure is a great place to support an R workshop or laboratory session or R training. I record here the simple steps to set up a Linux Data Science Virtual Machine (in the main so I can remember how to do it each time). Workshop attendees will have their own laptop computers and can certainly install R themselves but with the Ubuntu Data Science Virtual Machine we have a shared and uniformly configured platform which avoids the traditional idiosyncrasies and frustrations that plague a large class installing software on multiple platforms themselves. Instead of speding the first trouble filled hour of a class setting up everyone’s computer we can use a local browser to access either Jupyter Notebooks or RStudio Server running on the DSVM.

Jupyter Notebooks on JupyterHub

We illustrate the session with both Jupyter Notebook supporting multiple users under JupyterHub and as a backup running RStudio Server (for those environments where a secure connection through https is not permitted). Both can be accessed via browsers. JupyterHub uses https (encrypted) which may be blocked by firewalls within organisations. In that case an RStudio Server over http is presented as a backup.

WARNING: Jupyter Notebook has been able to render my laptop computer (under both Linux and Windows, Firefox and IE) unusable after a period of extensive usage when the browser freezes and the machine becomes completely unresponsive.

Jupyter Notebook provides a browser interface with basic literate programming capability. I’ve been a fan of literate programming since my early days as a programmer in the 1980’s when I first came across the concept from Donald Knuth. I now encourage literate data science and it is a delight to see others engaged is urging this approach to data science. Jupyter Notebooks are great for self paced learning intermixing a narrative with actual R code. The R code can be executed in place with results displayed in place as the student works through the material. Jupyter Notebooks are not such a great development environment though. Other environments excel there.

JupyterHub supports multiple users on the one platform, each with their own R/Jupyter process. The Linux Data Science Virtual Machine running on Azure provides these open source environments out of the box. Access to JupyterHub is through port 8000.

Getting Started – Create a Ubuntu Data Science Virtual Machine

To begin we need to deploy a Ubuntu Data Science Virtual Machine. See the first two steps on my blog post. A DS14 server (or D14_V2 for a SSD based server) having 16 cores and 112 GB of RAM seems a good size (about $40 per day).

We may want to add a disk for user home folders as they can sometimes get quite large during training. To do so follow the Azure instructions:

First time you connect to the site you will be presented with a warning from the browser that the connection is insecure. It is using a self signed certificate to assure the encryption between your browser and the server. That is fine though a little disconcerting. As the user you could simply click through to allow the connection and add an exception. This often involves clicking on Advanced and then Add Exception… and then Confirm Security Exception. It is safe to provide an exception for now. However, best to install a proper certificate!

Install a LetsEncrypt Certificate

We can instead install a free Let’s Encrypt certificate from letsencrypt to have a valid non-self-signed certificate. To do so we first need to allow connection through the https: port (443) through the Azure portal for the dsvm. Then log on to the server and do the following:

Log in to the server. This will depend on whether you set up a ssh-key or a username and password. We assume the latter for this post. On a terminal (or using Putty on Windows), connect as:

$ ssh xyz@dsvmxyz01.southeastasia.cloudapp.azure.com

You will be prompted for a password.

We can then create user accounts for each user in our workshop. The user accounts are created on the Linux DSVM. Here we create 40 user accounts and record their random usernames and passwords into the file usersinfo.csv on the server:

Notice a number of notebooks are available. Click the IntroTurorialInR.ipynb for a basic introduction to R.

Backup Option – RStudio

JupyterHub requires https and so won’t run internally within a customer site if they have a firewall blocking all SSL (encrypted) communications. In this case RStudio server is a backup option. It is pre-installed on the server and if you followed my instructions above for deploying a DSVM you will hav updated to the latest version too.

If you followed my DSVM deployment guide then you will have also set up X2Go on your local computer to support a desktop connection across to the DSVM. This is very convenient in terms of running desktop apparitions, like Rattle, on the DSVM. Every student in the class gets the same environment.

Shortcuts to the Services

The URLs are rather long and so we can set up either bit.ly or aka.ms shortcuts. Visiting the latter we set up two short URLs:

https://aka.ms/xyz_hub as https://dsvmxyz01.southeastasia.cloudapp.azure.com:8000
http://aka.ms/xyz_rstudio as http://dsvmxyz01.southeastasia.cloudapp.azure.com:878

We can now use the short URLs to refer to the long URLs.

REMEMBER: Deploy-Compute-Destroy for a cost effective hardware platform for Data Science. Deallocate (Stop) your server when it is not required.

Programming is an art and a way we express ourselves. As we write our programs we should keep in mind that someone else is very likely to be reading it. We can facilitate the accessibility of our programs through a clear presentation of the messages we are sharing.

As data scientists we also practice this art of programming. Indeed even more so we aim to share the narrative of our discoveries through our living and breathing of data through programming over the data. Writing programs so that others understand why and how we analysed our data is crucial. Data science is so much more than simply building black box analyses and models and we should be seeking to expose and share the process and particularly the knowledge that is discovered from the data.

Style is important in making the code we share readily accessible. Dictating a style to others is a sensitive issue. We thrive on our freedom to innovate and to express ourselves how we want but we also need consistency in how we do that and a style guide supports that. A style guide also helps us journey through a new language, providing a foundation for developing, over time, our own style in that language.

Through a style guide we share the tips and tricks for communicating clearly through our programs. We communicate through the language — a language that also happens to be executable by a computer. In this language we follow precisely specified syntax to develop sentences, paragraphs, and whole stories. Whilst there is infinite leeway in how we express ourselves in any language we can share a common set of principles as our style guide.

Over the years styles developed for very many different languages have evolved together with the medium for interacting with computers. I have a style guide for R that presents my personal and current choices. This is the style guide I suggest (even require) for projects I lead.

I hope the guide might be useful to others. It augments the other R style guides out there by providing the rationale for my choices. Irrespective of whether specific style suggestions suit you or not, choose your own and use them consistently. Do focus on communicating with others in the first instance and secondarily on the execution of your code (though critical it is). Think of writing programs as writing narratives for others to read, to enjoy, to learn from and to build upon. It is a creative act to communicate well with our colleagues — be creative with style.

The paper includes a discussion of turning ensemble concepts into the extreme, reflecting on the need for the pendulum to swing back toward protecting privacy, and the resulting focus on massively ensembled models, each “model” modelling an individual across extensive populations. Stay tuned for a future blog post that will explore this concept further and see the Research Gate project.

]]>Rattle 5.0.0 Alpha Released – ggraptR and Microsoft R Supporthttps://togaware.com/rattle-5-0-0-alpha-released/
Mon, 12 Sep 2016 05:21:41 +0000http://togaware.com/?p=574I have released an alpha version of Rattle with two significant updates.

Eugene Dubossarsky and his team have been working on a Shiny interface to generate ggplot2 graphics interactively. It is a package called ggraptR. This is now available through Rattle’s Explore tab choosing the Interactive option.

In line with Rattle’s philosophy of teaching programming of data by exposing all code through Rattle’s Log tab, ggraptR has a button to generate the plot. You can click the Generate Plot Code button, copy the resulting code and paste it into the R console, knitr document, or jupyter notebook. Execute the code and you generate the plot. Now you can start fine tuning it some more if you like.

The current alpha version has a few niggles that are being sorted out but it is already worth giving it a try.

The second major update is the initial support for Microsoft R Server so that Rattle can now handle datasets of any size. From Rattle’s Data tab choose an XDF file to load.

A sample of the full (generally big) dataset will actually be loaded into memory but many of the usual operations will be performed on the XDF dataset on disk. For example, build a decision tree and Rattle will automatically choose rxDTree() for the XDF dataset instead of rpart().

Visualise the tree as usual.

Performance evaluation is also currently supported.

Do check the Log tab to review the commands that were executed underneath.

This is an initial release. There’s still plenty of functionality to expose. Currently implemented for Binary Classification:

At an R event hosted by CSIRO in Canberra in 2011 Hadley noted that we are missing a grammar for machine learning. At the time I doodled some ideas but never developed. I repeat those doodles here. The idea’s are really just that – ideas as a starting point. Experimental code is implemented in the graml package for R which is refining the concepts first explored in the experimental containers package.

A grammar of machine learning can follow the ggplot2 concept of building layer upon layer to define the final model that we build. I like this concept rather than the concept of a data flow for model building. With a data flow a dataset is piped (in R using magrittr’s %>% operator) from one data wrangling step to the next data wrangling step. Hadley’s tidyr and dplyr do this really well.

The concept of a grammar of machine learning begins with recognising that we want to train a model:

train(ds, form(target ~ .))

Simply we want to train a model using some dataset ds where one of the columns of the dataset is named target and we expect to model this variable based on the other variables within the dataset (signified by the ~ .).

Generally in machine learning and statistical model building we split our dataset into a training dataset, a validation dataset, and a testing dataset. Some use only two datasets. Let’s add this in as the next “layer” for our model build.

train(ds, form(target ~ .)) +
dataPartition(0.7, 0.3)

That is, we ask for 70% of the data randomly sampled to train the model.

We will have already performed our data preparation steps and let’s say that we know in ds the target variable has only two distinct values, yes and no. Thus a binary classification model is called for.

In R we have a tremendous variety of model building algorithms that support binary classification. My favourite has been randomForest so let’s add in our request to train a model using randomForest().

]]>Data Science explained Simplyhttps://togaware.com/data-science-explained-simply/
Wed, 06 Jul 2016 04:31:21 +0000http://togaware.com/?p=492A 5-video series called Data Science for Beginners has been released by Microsoft. It introduces practical data science concepts to a non-technical audience… making data science accessible – keeping the language clear and simple as an entry point to understanding data science.

]]>The weatherAUS datasethttps://togaware.com/the-weatheraus-dataset/
Thu, 05 Nov 2015 10:42:01 +0000http://togaware.com/?p=439The R package rattle provides a dataset that I have been collecting over a few years now from the Australian Bureau of Meteorology. Like most of the datasets in rattle it is also available as a CSV file as part of the package (as well as a proper R dataset) and can also be downloaded from the Internet at http://rattle.togaware.com/weatherAUS.csv

The dataset has been sourced from the bureau since about 2008 for nearly 50 weather stations, some of which we can see on the map below which comes from the bureau:

Graham @ Togaware

]]>Rattle v4.0.0 Now on CRANhttps://togaware.com/rattle-v4-0-0-now-on-cran/
Thu, 05 Nov 2015 10:35:09 +0000http://togaware.com/?p=441A new release of Rattle has hit CRAN – this is version 4.0.0 and brings a variety of stability fixes and enhancements. For example, Jose A Magaña has added support for the display of pairs plots.

An obvious addition is the Connect-R button on the toolbar – this will take you to Connect-R where R related projects (including suggestions for enhancements to Rattle) can be listed and crow-funding applied to have the projects completed. Jose’s project to add pairs plots is an example of a crowd funded addition to Rattle.

Other enhancements include

more migration of plots to using ggplot2,

multiple ggplot2 plots within the single window,

add Group By to override the default group by the target variable for plots,