Nathan Bijnens

Introduction

Nathan Bijnens

I am Nathan Bijnens, a developer with a passion for great code,
Big Data and the Internet of Things. I am interested
in programming and system administration, especially
where they meet, from scaling platforms to designing the architecture of new and existing products and everything in between.
I am a passionate speaker and evangelist, on Big Data and
IoT.

I am focused on data analysis and building Big Data applications.
Using Spark, Storm, Hadoop, Cassandra and Kafka.
I am especially interested in real-timeBig Data, actively developing applications on top of Spark Streaming and Storm,
designing Lambda-like architectures.
I advise on Big Data Strategies and evangelizeBig Data and IoT to clients and at conferences.

I have a lot of experience with Java, Scala
and a general interest in Functional Programming.

I am a passionate Linux system engineer, follower of the DevOps movement, with experience in
Puppet, Chef,
NixOS and the Cloud.

I enjoy working with clients and partners,
from giving advice, talking about the
Business and Technological value of
Big Data and Internet of Things, to
Requirement Analysis.

I am inquisitive, I love learning about new things and
improving what I know. I am very passionate
about what I do, and I have strong analytical skills.

Presentations

Virdata: lessons learned from the Internet of Things and M2MCloud Services

at the

IBM Big Data Developers (San Jose, CA) Meetup of
June '14

.

A real-time (Lambda) architecture using Hadoop and Storm

at

NoSQL Matters Cologne '14

.

A real-timearchitecture
using Hadoop and Storm

at

Devoxx '13

, the biggest European Java conference.

A real-timearchitecture
using Hadoop and Storm

at

JAX
London '13

, a Java
Enterprise and Big Data conference.

Big Data, Hadoop and HDInsight

, together with Wesley Backelant of Microsoft Belgium
at the

SQLUser
Group Belgium

meetup of February 2013.

A Series of 3

Apache StormWorkshops

in the context of

BigData.be

, during 2013.

A
real-time architecture using Hadoop & Storm

at the2013 edition of the

FOSDEM
NoSqlRoom

, an open source conference, it has been viewed over 24500+ times.

A Vision on Data

at the2013

Big Data &
Security Conference

of

LSEC

.

Getting More out of Your Big Data

at

Microsoft Inspirience the Future 2012

.

Hadoop Pig: MapReduce the easy way

at the 2011 edition
of the

FOSDEM DataDevRoom

, an open source conference, it was featured on the front page of slideshare.net and
has been viewed over 18000 times.

experience

As from October 2013 I work for

Virdata

a start-up funded by

Technicolor

, as a

sr. Big Data and IoT Engineer

, previously as a

DevOps Engineer

. Virdata is an Internet of ThingsCloud platform, scalable to millions of devices, from tiny sensors to TVs to Cars.

I work on our Data Processing stack, on top of Spark and Spark Streaming, mostly in Scala. I also worked as the lead on our Sparkas a ServiceInfrastructure.

Data Engineer

From January 2014.

As a Big Data Engineer, I am working on our Data Processing and Analytics stack, creating a Lambda Architecture. Working with Spark, Spark Streaming, Storm, Cassandra, Kafka and HDFS. We mostly use Scala and bits of Java. We deploy using Chef on Mesos in the cloud.

Sparkas a Service Lead

From May 2014 till December 2014.

We created a Sparkas a Service,
where Data Scientists can on demand request a Spark and Mesos cluster in the Cloud.
We leveraged our existing Chef & Mesos architecture. We also added tools for ad hoc analytics, like IPythonPySpark & ScalaSpark notebooks. It is also possible to launch Docker containers.

Cloud & DevOps Engineer

From October 2013 till December 2013.

As a Big Data and DevOps Engineer, I was responsible for the scalability, quality and Operational Intelligence of a new platform. I introduced the Continuous Integration
platform, using Jenkins,
added unit tests. The platform is build using Chef,
tries to be cloud independent (runs on Amazon AWS,
Softlayer, ...).

Lambda Architecture

. The Lambda Architecture is a data-processing architecture designed to handle Big Data by combining both batch- and stream-processing, while attempting to balance latency, throughput, and fault-tolerance. Batch Processing is used to provide comprehensive and accurate (although slightly out of date) views, while simultaneously using real-timestream processing to bridge the gap.

As from April 2012 till March 2014 I worked for

DataCrunchers

, as a

Hadoop and Big Data Consultant

. DataCrunchers
is the leading Belgian consultancy firm around everything Big Data. I
co-developed on our internal Semantic Analysis Engine on top of Storm, in Java, I
co-presented our Hadoop Ecosystem course, I created a new
website using Drupal and I was responsible for our MicrosoftBig Data partnership.

Big Data & DevOps Engineer,
Technicolor

From October 2013.

As a Big Data and DevOps Engineer, I was responsible for the scalability, quality and Operational Intelligence of a new platform. I introduced the Continuous Integration
platform, using Jenkins,
added unit tests. The platform is build using Chef,
tries to be cloud independent (runs on Amazon AWS,
Softlayer, ...).

Project Manager ABBAMelda, Flemish Government (Macq)

From June 2013
till October 2013.

Project Manager and Lead Developer of ABBAMelda, a ticket and maintenance management system, originally developed by Siemens.
ABBAMelda consists of a JavaEE backend, an Informix
database, with a PHP/jQuery frontend. Under my lead, we created an enhanced Tablet intranet site, improved the
bulk upload possibilities, additional REST services, introduced unit
testing and switched to git. I was responsible for coordinating with different teams within
the Flemish Government, as well as the contractor (Macq).

A social media startup

From February 2013 till June 2013.

Defining and implementing the architecture
for a social mediaanalytics
startup. I designed and implemented a Lambda Architecture (in Java), on
top of Storm and Hadoop, using Redis, Voldemort as well as Thrift.

HSHMRK

From December 2012 till January 2013.

Defining and implementing the architecture for hshmrk, a
data visualization startup. The application backend is written as a
JerseyREST (Java),
service, using ElasticSearch as
storage. The frontend is an AngularJS and D3
web application. This approach allowed us to easily scale.

iSubscriber

Creating a small web application to organize
and input subscriptions into Octopus.

MySQLPHPsymfony

Various clients

From July till November 2011.

Analyzing the requirements, estimating and taking the lead.

Extracting data from various Accounting software to
iController.

Accounting SoftwareAS400ETLXMLXSchemaOracleMSSQLNavision

LeasePlan

From July till November 2011.

Analyzing the requirements, coaching my co-developers and coordinating with the
client.

Extracting data from and linking SAP FI to
iController.

Developing a custom workflow engine.

SAP FIAS400ETLXMLSQL

From September
2010 till February 2011 I was employed at

Netlog

as a

Warehouse and Web Developer

,
creating an analytical warehouse based on Hbase. I also created large parts of the
processing infrastructure using Hadoop and Hadoop Pig. I
also advised on
new technologies, git migration, best practices
& unit tests. Netlog is a social community network, with over
70 million members, mainly active in Europe and the Arabic world.

Warehouse Developer

Setting up the Hadoop infrastructure, analyzing data
with Hadoop Pig
and creating a dashboard using Symfony2 and Hbase &
Thrift.

Twoo Core Framework Developer

Creating a new PHP Framework (no existing frameworks were
allowed) as base for
the new dating site Twoo. I advised on Design Patterns
and Best
Practices. I developed the Security and
ACL platform as well.

Web Developer (Netlog)

I introduced an open source Event framework
and
presented it to my
co-developers. I introduced new application log functionality, for logging to
Hadoop. I evangelizedUnit tests,
including an initial implementation
and presentations. I evaluated Redis, Memcached & Membase
(now Couchbase).

Truvo

hadoop deduplication

job. I used Hadoop & Hadoop Pig, wrote custom UDFs in
Java for Ngram matching and solved
performance
issues. At first Amazon Elastic
MapReduce was used, later I setup a Hadoop cluster.

HadoopHadoop PigCloudAmazon EC2Amazon Elastic MRJava

In 2009 I was active as a consultant to a Dutch services organization. The assignment mainly consisted of

creating a link between existing systems and a new website

.

From 2008 till present I perform various

short term consultancy

tasks commissioned by

Sinergio

. For a

semi government organization

I set up a new tomcat server and
did a general check up
on their Linux servers. For another company I configured an ASP.Net application and installed
an IIS server. For

Sio Hosting

(Formerly Sinergio Hosting) I planned and
executed the move
from traditional servers to a virtualizedxen cluster.
I also checked the integrity and security
of their hosting platform on a regular basis. I advised them on
strategic planning.

During my secondary education I created a

PHP CMS

, with features as native PHP templates,
module in module support (recursion) following a very basic MVC pattern. It was
used and expanded by
two web agencies to create over 30 websites.

skills

I am very interested in and work with

Big Data

, from the processing and storage of large
volumes, to
real-timestream
processing, and machine learning. I read,
tweet, and
try out as much as I can about new Big Data technologies,
like Spark, Samza, Storm and Kafka,
as well as more established technologies like Hadoop and Cassandra and learning as
much as I can in the process.
I use Scala, Java and bits of Python and Hadoop Pig Latin. I setup, administered and monitored Hadoop,
Mesos, Storm, Cassandra & Zookeeper clusters.

Big Data related skills:

Do interactive ad-hoc analytics using Python based IPython Notebooks, as well as Scala based notebooks, combined with Spark SQL.

Combining batch and real-time
technologies to create a Lambda
architecture (of Nathan Marz), that is resilient to failure, scalable
and fast.

Next to programming I have always been passionate about

Linux

and open source. I have
used over the last 10+ years several distributions from Debian
& Ubuntu,
CentOS over Gentoo to Linux From Scratch, lately I am very interested in NixOS. NixOS is a purely functional Linux. I have setup countless
servers from Cloud to virtualized to bare metal servers.

I am following the DevOps movement. I am using Puppet, Chef and NixOps to automate and Ganglia to monitor critical infrastructure. I have open sourced
and contributed to several NixOS and Puppet modules.

I follow and try out with great interest Cloud related techniques
and technologies, in all its forms: IaaS, PaaS, MaaS, … I have used as test or in production OpenStack, Amazon S3, Amazon EC2,
Amazon MapReduce, Amazon
IAM, Google
BigQuery2 (private beta tester), Windows Azure
Platform and Hadoop on Azure (private beta tester) and Softlayer.

I currently mostly develop using

Scala

. Scala is a JVM-based language, with both full support for OO as well as Functional Programming approach. I started with Scala at the end of 2013.

Using Scala to create Spark and Spark Streaming jobs.

Using Akka and Akka Persistence to create a CQRS application, in Scala.

Creating Thrift based services, using Finagle.

Unit testing using Scalatest and Cucumber.

I am interested in

Functional Programming

, mostly looking at Scala, and a bit at Clojure. Especially functional programming in relation to
Big Data has my focus.
I am using

Java

, mostly using Spring, maven and Jersey in
combination with the JavaScript MVC framework AngularJS.

Developing a Java EE application, with Glassfish, using
JPA with EclipseLink.

Creating Threaded servers, using Thrift.

Autowiring & Dependency Injection using Spring.

Consuming the Twitter & LinkedInAPIs, using
OAuth.

Creating a REST and SOAP based
services, using Jersey or JAX-WS.

Using the common libraries like Guava, Apache Commons, Joda Time
& slf4j.

Unit testing using JUnit and Mockito.

I have used

PHP

and

JavaScript

for over 10 years,
mostly in a combination with the symfony & Symfony2
frameworks as well as a multitude of SQL and NoSQL databases, as
MySQL, PostgreSQL, Oracle,
Informix
and Redis, Membase, Memcached and ...

html5, CSS & jQuery

Creating a Drupal site, using Bootstrap and deploying on Windows Azure.