Between some of the games I had a go at replicating a plot from liberation.fr on the connections between Euro 2016 players and the country of birth using the circlize package in R. As with the previous post, the colours are based on the home shirt of each team and data scraped from Wikipedia. The values in the parenthesis represent the total number of players born in the respective country, which dictates their ordering around the circle. It is interesting to see just how many players represent countries that they were not born in. Only Romania has a 23 man squad completely full of players born in the country and no other Romanian born players representing other nations.

I also modified some the my ggplot2 mapping code in my previous post to plot the links between players place of birth and the capital city of their national team: Click on the images if you want to explore. Here is an expanded view to illustrate all the longer distance relationships: All the code is up on my Github.

This weekend I was having fun in France watching some Euro 2016 matches, visiting friends and avoiding Russian hooligans. Before my flight over I scraped some tables on the tournaments Wikipedia page with my newly acquired rvest skills, with the idea to build up a bilateral database of Euro 2016 squads and their players clubs.

On the flight I managed to come up with some maps showing these connections. First off I used ggplot2 to plot lines connecting the location of every players club teams to their national squads base in France. The path of the lines were calculated using the gcIntermediate function in the geospherepackage. The lines colour is based on the national teams jersey, which I obtained via R using the amazing extract_colours function in the rPlotterpackage.I was not entirely convinced that this plot is too effective as the base camp of each team in France is not particularly common knowledge. This led me to create a new plot with a link from the players club teams to their nations capital city.This shows some clear relationships, such as between the Iceland players and clubs in Scandinavia and clubs in northern England and Scotland with the Irish teams. Here is another plot to show some of the more distant relationships for players from non-European clubs

Finally, I had a go at trying to match my old World Cup circular plot using the chordDiagram function in the circlizepackage.

I ordered the countries according to their UEFA coefficient, which is based on the performance of club teams in European competitions. Most players playing abroad are based in teams in the top leagues of England, Germany, Italy or Spain. Players based in the English leagues make up most of the squads for teams from the British Isles. There are sizeable numbers of the Austrian and Swiss squads playing for clubs in the German leagues and Croatian players in Italy.

All the code for the scraping the data and producing the plots are on my Github. Check out my next post for more on the players at Euro 2016.

I have had a few emails recently regarding plots from my new working paper on global migration flows, which has received some media coverage here, here and here. The plots were created using Zuguang Gu’s excellent circlize package and are modified version of those discussed in an earlier blog post. In particular, I have made four changes:

I have added arrow heads to better indicate the direction of flows, following the example in Scientific American.

I have reorganized the sectors on the outside of the circle so that in each the outflows are plotted first (largest to smallest) followed by the inflows (again, in size order). I prefer this new layout (previously the inflows were plotted first) as it allows the time sequencing of migration events (a migrant has to leave before they can arrive) to match up with the natural tendency for most to read from left to right.

I have cut out the white spaces that detached the chords from the outer sector. To my eye, this alteration helps indicate the direction of the flow and gives a cleaner look.

I have kept the smallest flows in the plot, but plotted their chords last, so that the focus is maintained on the largest flows. Previously smaller flows were dropped according to an arbitrary cut off, which meant that the sector pieces on the outside of the circle no longer represented the total of the inflows and outflows.

Combined, these four modifications have helped me when presenting the results at recent conferences, reducing the time I need to spend explaining the plots and avoiding some of the confusion that occasionally occurred with the direction of the migration flows.

If you would like to replicate one of these plot, you can do so using estimates of the minimum migrant transition flows for the 2010-15 period and the demo R script in my migest package;

The code in the demo script uses the chordDiagram function, based on a recent update to the circlize package (0.3.7). Most likely you will need to either update or install the package (uncomment the install.packages lines in the code above).

If you want to view the R script in detail to see which arguments I used, then take a look at the demo file on GitHub here. I provide some comments (in the script, below the function) to explain each of the argument values.

Save and view a PDF version of the plot (which looks much better than what comes up in my non-square RStudio plot pane) using:

A few weeks ago a new version of the the Wittgenstein Centre Data Explorer was launched. The data explorer is intended to disseminate the results of a recent global population projection exercise which uniquely incorporates level of education (as well as age and sex) and the scientific input of more than 500 population experts around the world. Included are the projected populations used in the 5th assessment report of the Intergovernmental Panel on Climate Change (IPCC).

Over the past year or so I have been working (on and off) with the data lab team to create a shiny app, on which the data explorer is based. All the code and data is available on my github page. Below are notes to summarise some of the lessons I learnt:

1. Large data

We had a pretty large amount of data to display (31 indicators based on up to 7 scenarios x 26 time periods x 223 geographical areas x 21 age groups x 2 genders x 7 education categories)… so somewhere over 8 million rows for some indicators. Further complexity was added by the fact that some indicators were by definition not available for some dimensions of the data, for example, population median age is not available by age group. The size and complexity meant that data manipulations were a big issue. Using read.csv to load the data didn’t really cut the mustard, taking over 2 minutes when running on the server. The fantastic saves package and ultra.fast=TRUE argument in the loads function came to the rescue, alongside some pre-formatting to avoid as much joining and reshaping of the data on the server as possible. This cut load times to a couple of seconds at most, and allowed the app to work with the indicator variables on the fly as demanded by the user selections. Once the data was in, the more than awesome dplyr functions finished the data manipulations jobs in style. I am sure there is some smarter way to get everything running a little bit quicker than it does now, but I am pretty happy with the present speed, given the initial waiting times.

2. googleVis and gvisMerge

It’s a demographic data explorer, which means population pyramids have to pop-up somewhere. We needed pyramids that illustrate population sizes by education level, on top of the standard age and sex breakdown. Static versions of the education pyramids in the explorer have previously been used by my colleagues to illustrate past and future populations. For the graphic explorer I created some interactive versions, for side-by-side comparisons over time and between countries, and which also have some tool tip features. These took a little while to develop. I played with ggvis but couldn’t get my bar charts to go horizontal. I also took a look at some other functions for interactive pyramids but I couldn’t figure out a way to overlay the educational dimension. I found a solution by creating gender specific stacked bar charts from gvisBarChart in the googleVis package and then gvisMerge to bring them together in one plot. As with the data tables, they take a second or so render, so I added a withProgress bar to try and keep the user entertained.

I could not figure out a way in R to convert the HTML code outputted by the gvisMerge function to a familiar file format for users to download. Instead I used a system call to the wkhtmltopdf program to return PDF or PNG files. By default, wkhtmltopdf was a bit hit and miss, especially with converting the more complex plots in the maps to PNG files. I found setting --enable-javascript --javascript-delay 2000 helped in many cases.

3. The shiny user community

I asked questions using the shiny tag on stackoverflow and the shiny google group a number of times. A big thank you to everyone who helped me out. Browsing through other questions and answers was also super helpful. I found this question on organising large shiny code particularly useful. Making small changes during the reviewing process became a lot easier once I broke the code up across multiple .R files with sensible names.

4. Navbar Pages

When I started building the shiny app I was using a single layout with a sidebar and tabbed pages to display data and graphics (using tabsetPanel()), adding extra tabs as we developed new features (data selection, an assumption data base, population pyramids, plots of population size, maps, FAQ’s, etc, etc.). As these grew, the switch to the new Navbar layout helped clean up the appearance and provide a better user experience, where you can move between data, graphics and background information using the bar at the top of page.

5. Shading and link buttons

I added some shading and buttons to help navigate through the data selection and between different tabs. For the shading I used cssmatic.com to generate the colour of a fluidRow background. The code generated there was copy and pasted into a tags$style element for my defined row myRow1, as such;

I added some buttons to help novice users switch between tabs once they had selected or viewed their data. It was a little tougher to implement than the shading, and in the end I need a little help. I used bootsnipp.com to add some icons and define the style of the navigation buttons (using the tags$style element again).

That is about it for the moment. I might add a few more notes to this post as they occur to me… I would encourage anyone who is tempted to learn shiny to take the plunge. I did not know JavaScript or any other web languages before I started, and I still don’t… which is the great beauty of the shiny package. I started with the RStudio tutorials, which are fantastic. The R code did not get a whole lot more complex than what I learnt there, even though the shiny app is quite large in comparisons to most others I have seen.

I have been having a go in R at visualising player movements for the World Cup. I wanted to use similar plots to those used to visualise international migration flows in the recent Science paper that I co-authored. In the end I came up with two plots. The first, and more complex one, is based on a non-square matrix of leagues system of players clubs by their national team.

You can zoom in and out if you click on the image.

Colours are based on the shirt of each team in the 2014 World Cup. Lines represent the connections between the country in which players play their club football (at the lines base) and their national teams (at the arrow head). Line thickness represent number of players. It’s a little cluttered, but shows nicely how many players in the English, Italian, Spanish and French leagues are involved in the world cup. It also highlights well some countries where almost all the players are at clubs abroad, for example most of the players in the African squads.

Whilst the first plot gave a lot of detail, I wanted to visualise the broader interactions, so I aggregated over leagues systems and national squads by regional confederations. This gives a square matrix:

This type of aggregation works really well to show how few European national players play elsewhere (only Zvjezdan Misimovic in all the European World Cup squads). It also provides a way to compare the share of non-European players plying their trade in the European leagues to those in more local leagues within their confederation.

I scraped the data from the provisional squads on Wikipedia, and then created the images with the circlize package. All the code to reproduce the plots + scraping the Wikipedia squad pages are on the my github.