If you recently tried to go to the main Dex site and saw GoDaddy Cash Parking advertisements instead; here is why:

GoDaddy apparently has set up a program for domain squatters to gain revenue from previously active domains. The domain owner (in my case a squatter who pounced on my domain once GoDaddy failed to auto-renew it) then makes revenue off of a parked site based upon advertisement. In turn, GoDaddy gets a nominal monthly fee for the Cash Parking service.

Basically, they create a market for this bad behavior.

So I switched my registrar and the domain which ties to the Dex project.

Share this:

Like this:

There’s lots of talk nowdays about the cloud, containerization, so I decided to see what it would take to run a JavaFX application within a docker container.

It took a bit of work to figure out, but it works great.

Due to some Oracle licensing requirements the version of Oracle Java 8 is not bundled like normal linux packages, but rather has to be downloaded and installed from a special repository called webupd8. This small amount of additional complexity is also reflected in the Dockerfile we will cover below.

Why Docker?

Why bother bundling an application with Docker? Well, there are many reasons.

Reproducible installations — Since docker ships a machine with the application, if it runs on the source application container, it’s almost guaranteed to run on the target environment.

It would be nice to break it up some, but there’s a good reason that we don’t. Multiple RUNs within a Dockerfile will cause the creation of multiple layers; one per RUN. We’re trying to minimize this, and the objective causes some ugliness. It’s necessary though.

Basically, this command is performing the following task:

Lines 19-23 are setting up some repositories.

Line 24 updates the packages.

Line 25 installs a few necessary dependencies required for JavaFX libprism dependencies.

Lines 29-30 automate the acceptance of conditions and lines 31 performs a headless installation of Java8.

Line 33-34 installs git then uses it to download the Dex repository, code and all to the current directory.

Lines 36-39 perform some cleanup.

Line 42 specifies the command we will run by default when executing this application container.

Building from Dockerfile

Assuming docker is installed and running, drop the Dockerfile described above into an empty directory somewhere.

Issuing the command:

docker build -t dex .

Will create a dex image. It’ll take a few minutes to run. Remember, it’s creating a vm image, updating it to spec, installing oracle and dex. That’s a lot!

Share this:

Like this:

Grunt/Gulp are Overkill (for me)

I have come to the conclusion that specialized wrapper/runner systems such as Grunt and Gulp are unnecessary technical debt. I had previously been using Gulp and all the pertinent Gulp add-ons that I needed and it was a source of complexity.

I have my main 4 task defined as scripts. Within WebStorm I can execue individual scripts with a single button click. It’s minimalistic, though script strings can get unwieldy. Here are my new set of dependencies. I now have 7 package dependencies spread across 6,750 files taking up a total of 45M of disk space. That’s much better than my previous 54 node packages spread across 31,239 files taking up 343M of disk space.

Share this:

Like this:

Lately I’ve been droning on about the reusable player and the merits of being able to create frames of data which may be played out across a dimension (typically time). I think the first time I really saw this in action was watching Hans Rosling present “200 Countries, 200 Years, 4 Minutes – The Joy of Stats”. To me, it looked like science fiction; at first glance.

Being an engineer, I condensed this in my head as 200 series, consisting of 200 samples with an x and a y value. Roughly 200 x 200 x 2 datapoints depending on sparseness of the sampling. Roughly 80,000 data values represented by 40,000 data points presented in manner understandable by virtually anyone. 40,000 points on a screen would look like chaos and would be difficult from which to derive anything insightful. Removing a dimension via framing or partitioning; in this case the dimension of time, reduces the number of points on the screen at any given time to a much more manageable 200; albeit this is still a lot for the human brain to handle. Series selection and filters come to the rescue.

One of the downsides of partitioning, or framing out a dimension is that you may only view one frame at a time. Something that Gapminder solved by adding trails to selected series. Literally providing the best of both worlds. A tremendous reduction in visual clutter without loss of signal. The ability to show multiple trails allows for direct comparison between two series.

So the other day, serendipity struck in the form of a tweet which informed me of a pre-alpha project called Vizabi, a Javascript project on Github which leverages D3js to provide those wonderfully insightful Gapminder charts. Aha! The geniuses at Gapminder are sharing their secrets at long last! I’m looking forward to peeking under the hood. Anyone reading this should check it out.

Child Mortality & GDP per capita

This is a compelling view within Gapminder because it relates prosperity to quality of life. There is no greater common instinct in human beings more powerful than protecting our children. This is a great view on the overall health and direction of a nation.

Click the image to see for yourself.

So, at a glance of the snapshot of 2015 affairs, we see a great disparity between the light blue and the rest of the world. Picking a nation, “Burundi”, I enable trail and look for major points of inflection.

In 1945, child mortality starts dropping at a more accelerated pace. Cross referencing Wikipedia, significant events within this time frame were:

1940 : Classified as a United Nations Trust Territory under Belgian administrative authority.

1943: Powers were split in the legislative division of Burundi’s government between chiefdoms and lower chiefdoms. Chiefdoms were in charge of land, and lower sub-chiefdoms were established. Native authorities also had powers.

In 1960, we start seeing a decline in financial prosperity, while mortality rates fluctuate slightly. Again, according to Wikipedia, significant events in this time frame were:

1959: On January 20, 1959, Burundi’s ruler Mwami Mwambutsa IV requested Burundi’s independence from Belgium and dissolution of the Ruanda-Urundi union. In the following months, Burundian political parties began to advocate for the end of Belgian colonial rule and the separation of Rwanda and Burundi.

Don’t read this wrong, these are just the facts Jack! Perhaps not enough of the facts at that. I am an advocate of independence. However, it seems that there was a financial price for Burundi’s independence which took a decade to recover from financially. Family income declines from $632.00 to $531.00 with a gradual recovery over a 10 year period of time.

In 1974, the trend of increased financial prosperity became a decline until 1991 when things again started to turn around. Wikipedia, what happened?

1992: The government promulgated a new constitution that provided for a multi-party system. Civil war broke out.

1993: A first attempt at democracy. June 1993, Melchior Ndadaye won the first democratic election and became the first Hutu head of state, leading a pro-Hutu government before being assassinated in October of 1993. 300,000 people were killed in the following few years, but it looks like very few of them were 5 and under.

With so many unregistered deaths, obviously, a single view is not a complete view.

Finally, 2005 brings what looks to be a decade of decreasing Child mortality rates with an increase in prosperity (though very much poverty level on a global scale).

Trans-formative events around this time frame appear to be:

2000: On 28 August 2000, a transitional government for Burundi was planned as a part of the Arusha Peace and Reconciliation Agreement. The transitional government was placed on a trial basis for five years. After several aborted cease-fires, a 2001 peace plan and power-sharing agreement has been relatively successful

2003: A cease-fire was signed in 2003 between the Tutsi-controlled Burundian government and the largest Hutu rebel group, CNDD-FDD (National Council for the Defense of Democracy-Forces for the Defense of Democracy). Domitien Ndayizeye (Hutu) is elected president.

2005: Ethnic quotas are formed for determining positions in Burundi’s government. Pierre Nkurunziza (Hutu), once a leader of a rebel group, elected president.

2006: UN reconstruction efforts seem to take effect.

The Tutsi/Hutu Conflict

I never expected to learn so much about Burundi when I started this post. I didn’t realize that it was another country riddled with Tutsi/Hutu conflict. What’s all that about anyway? So I google:

500-1000 BC : The Hutus, an agricultural people, settle in the Great Lakes region of Central Africa.

1600 AD: The Tutsis, also known as Watutsis, a nomadic people, begin arriving from Ethiopia. They settle peacefully among the Hutus – adopting their language, beliefs and customs.

Gradually…economic differences form… Tutsis as cattle herders often being in a position of economic dominant position over the soil-tilling Hutus. As the Tutsis gain more dominance, the minority begins to rule the majority. Resentments form…

Over time, the Tutsi and Hutus has evolved into more of an economic difference rather than an ethnic one as explained by Charlayne Hunter Gault, Professor Izangola:

“In Rwanda, the Tutsi and the Hutu are the same people. They are all people–large grouping or communities which go from seven regions of Cameroon to Uganda–all the way to South Africa, in the same culture,” Izangola said. “People used to be Tutsi or Hutu, depending on the proximity to the king. If you were close to the king, you owned wealth, you owned a lot of cattle, you are a Tutsi. If you are far away from the king, you are a cultivator, you don’t own much cattle, you are a Hutu.”

Colonial rule, which began in the late 19th Century, did little to bring the groups together. The Belgians, who ruled what would later become Rwanda and Burundi, forced Hutus and Tutsis to carry ethnic identity cards. The colonial administrators further exacerbated divisions by only allowed Tutsis to attain higher education and hold positions of power.

Share this:

Like this:

Previously, I posted a visual of a single time-series against a global heatmap based upon the World Development Indicator dataset from the World Data Bank.

So I wondered, could I display all the series from this dataset? This is no small task as it consists of over 18 million datapoints spread across 1,345 series for 248 different countries and country aggregations.

Having had my site hacked a number of times, rather than worry about exposing a restful service to the internet, I decided to use an in-memory database. Sure the initial load will be slow, however, exploration itself would be fast.

Initially, it took me over a day to load the datapoints into sqlite. However, I didn’t get it right the first time. As Larry Wall once said, Impatience is an virtue of the developer. Impatient, I tried to figure out why my insertion rate into sqlite was only 211 entries per second. Turning autocommit off and committing every 1000 insertions increased my performance by roughly 4000 times. Once that change was made, my 24 hour load times became 3 minutes.

The schema I settled on looks like:

On my local machine, it takes around 8 seconds to initialize the application. From there, I can explore all 1,345 development statistics across time in a Gapminder like setting.

For a limited time, I will keep this link active so that others may also give it a try:

However, due to the fact that the sqlite database is over 500 megs (mostly due to indexing), I will likely take this offline in the near future.

-rwxrwxr-x+ 1 Patrick Patrick 423095296 Mar 6 19:49 wdi.db

Do not click this link from a mobile device! It will surely fail and expect it to take awhile to load! While the banner: “Please wait, the data is loading…” is displayed; THE DATA REALLY IS STILL LOADING.

Share this:

Like this:

In my last blog article I discussed the reusable Player component. So I’ve decided to put it to work on a real WDI dataset consisting of 5,166 rows. This example plays a heatmap of % users on the internet mapped across country.

Images link to live examples.

Tying the plot.ly chloropleth to the player was quite easy. In this case, I didn’t even bother to wrap the component in the Dex component framework, though I may do so in the future for consistency.

Next, I threw the same dataset at the C3 based player discussed in the earlier blog. I also expanded the player to support categorical x-axis. By framing the data in different ways, we can create very different views of the data and gain deeper insight.

In this example, I frame the data by country. The player plays the internet adoption across each country from 1989-2014. Here, we can see Afghanistan’s slow but increased adoption of the internet.

By simply reframing the datset by year, we can see all countries side by side. The player cycles through time. There are many labels, so the zoom and pan options of C3 come in handy here. Same data, but quite a different view.

Anyway, that’s it for now, but I expect that there will be much more to come.