Gianluca Arbezzano - GianArb2018-12-02T09:58:03+00:00http://gianarb.itGianluca Arbezzanohttp://gianarb.itgianarb92@gmail.comReactive planning is a cloud native pattern2018-11-28T08:08:27+00:00http://gianarb.it/blog/reactive-planning-is-a-cloud-native-pattern<p>Probably this title can sound a bit weird to anyone that already know what
reactive plan is and how far it can look from all the cloud-native and
distributed system hipster movement but recently one of my colleagues <a href="https://twitter.com/goller">Chris
Goller</a> pushed this pattern to one of the projects
that we have at <a href="https://influxdata.com">InfluxData</a> and I find it glorious!</p>
<p>“In artificial intelligence, reactive planning denotes a group of techniques for
action selection by autonomous agents. These techniques differ from classical
planning in two aspects. First, they operate in a timely fashion and hence can
cope with highly dynamic and unpredictable environments. Second, they compute
just one next action in every instant, based on the current context.”
(<a href="https://en.wikipedia.org/wiki/Reactive_planning">Wikipedia</a>)</p>
<p>The Wikipedia definition of reactive planning as you can see is perfect to
handle a system where the current status can change very frequently based on
external and unpredictable events.</p>
<p>This is a perfect approach for provisioning/orchestrator tool like Mesos, Cloud
Formation, Kubernetes, Swarm, Terraform. Some of them are working like this
already.</p>
<p>The general idea is that before any action you need a plan because for these
tools an action means: cloud interaction, spin up of resources that cost money.
You need to be proactive avoiding useless execution.</p>
<p>A plan is made of a serious of steps and every step can return other steps if it
needs. The plan is complete when there are no steps anymore. The plan gets
executed at least twice, the second time it should return zero steps because the
first attempts built everything you need, this is the signal that determines its
conclusion. If it keeps returning steps it means that there is something to do
and it tries again.</p>
<p>Let’s start with an example. Think about what Cloud Formation does. You can
declare a set of resources and before to take action it needs to understand what
to do. It is making a plan checking the current state of the system. This first
part makes the flow idempotent and solid because you always start from the
current state of the system. It doesn’t matter if it changes over time because
of somebody that removed one of the resources. If something doesn’t exist it
creates or modify it. Very solid.</p>
<p>Every single step is very small. Let’s take another example like creating a pod
in Kubernetes. When you create a pod there are a lot of actions to do:</p>
<ul>
<li>Validation</li>
<li>Generate the pod id, the pod name</li>
<li>Register the pod to the DNS, you</li>
<li>Store it to etcd
Reach to CNI to configure the network</li>
<li>Reach to docker, container or whatever you use to get the container</li>
<li>Maybe reach to AWS to create a persistent volume</li>
<li>Attach the PV</li>
</ul>
<p>If you try to design all interaction in a single “controller” you will end up
with a lot of * if/else, error handling and so on. Mainly because as you can see
almost every step interact over the network with something: database, DNS, CNI,
docker and so on. So it can fail, it needs circuit breaking, retry policy and
much more complexity.</p>
<p>It is a lot better to design the code where every point is a small step if the
step that reaches docker fails it can return itself as “retry” or it can return
other steps to abort everything and clean up. You will end up with small
reusable (or not that much reusable) steps.</p>
<p>All the steps are combined within a plan, the “PodCreation” plan. There is a
scheduler that takes and execute every step in the plan recursively.</p>
<blockquote>
<p>This freedom allows you to use an incremental approach</p>
</blockquote>
<p>The scheduler as first call a create method for the plan, the create method
checks what to do based on the current state of the system, it is the
responsibility of this function to return no steps when there is nothing to do.</p>
<p>I think this Reactive Planning is one of the best ways to organize the code in a
cloud-native ecosystem for its reactive nature as I said and for the fact that
it forces you to check the state of the system if you don’t do that the plan
will keep executing forever. Obviously, you can use a high-level check to skip
a lot of steps, this requires a balance if the plan you are executing is
critical and frequently used you should check for every step if it requires an
effort that won’t pay back you can implement deepest and preciser checks. You
can check for the PodStatus. If it is running we are good nothing to do. Or you
can check if Docker has a container running and if it has the right network
configuration. If it is running but with no network, you can return the step
that interacts with CNI to set the right interface. This freedom allows you to
use an incremental approach, you start with an easy creation method with checks
for only critical and high-level signal demanding a more solid and sophisticated
set of checks for later, when you will have the best knowledge about where
the system fails.</p>
<p class="small">Hero image via
<a href="https://pixabay.com/en/time-time-management-stopwatch-3222267/">Pixabay</a></p>
You will pay the price of a poor design2018-11-10T08:08:27+00:00http://gianarb.it/blog/price-of-poor-software-design<p>The unique way to avoid the price of a poor design project to replace them in
time. But we all know that replacing a software is not a great idea.</p>
<p>Software should be improved via multiple iterations and that’s why you or your
teammates will read a lot more code compared with the amount written and the of
the story.</p>
<p>In a rush of writing a new project, this sentence will sound wrong, but this is
just one of the phase of it. After that, it will be all about reading and
replacing lines of code one by one and if you poorly design the software
somebody will pay the price of it. You or somebody else, probably somebody else
looking at how quick a developer changes job.</p>
<blockquote>
<p>it is in your hands as a developer to think about design, at
least to save yourself from the darkness</p>
</blockquote>
<p>In a fancy unicorn startup, you will hear that there is no time to think about
design, if you are in tech probably that’s not true, but there is always
a project that nobody cares but it needs to be fixed.</p>
<p>Some other companies don’t have time to think about design because they are
always running against something.</p>
<p><img src="/img/hero/poverty.jpg" alt="" class="img-fluid" /></p>
<p>So, as you can see, it is in your hands as a developer to think about design, at
least to save yourself from the darkness.</p>
<p>Recently I read <a href="https://www.amazon.com/Philosophy-Software-Design-John-Ousterhout/dp/1732102201/">“Philosophy Software
Design”</a>
by John Ousterhout. This book turned out to be <strong>not</strong> a breath of fresh air <strong>for
me</strong> was more like a “breath of consolidation”. Professor John Ousterhout fixed on
paper, in an excellent way what I try to do every day but I was not always able to express.</p>
<p>Why comments are essential, deep API vs shallow classes, information hiding. You
should read it!</p>
<p><strong>Design it twice</strong>. It looks expensive but it is a take away from the book that
I think is the practical key to unlock the door to make our work fun and
healthy.</p>
<p>The first solution can’t be the best one. Even if you are smart enough to design
something that it won’t crash we should make an effort to think about another
solutions, to ask for a review, just as we do for when writing code.</p>
<p>In theory, we will have a great design looking in between of all the other
attempts.</p>
Chaos Engineering2018-08-23T08:08:27+00:00http://gianarb.it/blog/chaos-engineer<p>At the <a href="https://jazoon.com/">Jazoon</a> conference in Switzerland, I had the chance
to speak at the Chaos Engineering panel with <a href="https://twitter.com/russmiles">Russ
Miles</a> from <a href="https://chaosiq.io">ChaosIQ</a> and
<a href="https://twitter.com/aaronblohowiak">Aaron P Blohowiak</a> from Netflix.</p>
<p>The organizers put me in the panel probably because “chaos” was part of the title
for the talk I just gave in the morning. I was too curious to mention that I
never did it before, at least on purpose!</p>
<p>So I was really out of my comfort zone dealing with these two folks that know
their shit so well!</p>
<p>I am sure that as Engineers we are part of the Chaos, we create entropy inside the
system during every deploy and even if we have all the tests in the world the
first time it is tough to make it work. But I indeed never associated the
word engineering to chaos. And that’s the real challenge.</p>
<p>So, let’s define Chaos and Engineering altogether.</p>
<p>Let’s start with <code class="highlighter-rouge">Chaos</code> because it is the easy one, as I said we as developers create
chaos, distribution creates chaos, and customers create chaos. If somebody tells you
that his production environment is excellent, you should not listen to him,
Production is a nightmare, complicated and painful place. At least if somebody uses it.</p>
<p>And if it is just a bit more complicated than a static site it never works 100%,
the chaos governs it, and that’s where the sentence Engineering becomes
essential.</p>
<p><code class="highlighter-rouge">Engineering</code> at least for what I can understand means to be driven by data and
not feeling. So associating these two concepts together you have a powerful way
to measure the chaos.</p>
<p>I think you can’t avoid chaos, so the best way to handle it is to learn from
what it generates in your system to anticipate unpredictable situations.</p>
<p>As developers, ops or devops we are pessimistic about our system, and we know
that it will fail: servers crashes, CoreOS auto updates itself, third party
services stop to work. Usual the answer is to wait for it to happen usually
Friday night.</p>
<p>Chaos Engineering is an exercise, a practice to leverage “unusual but possible”
situations as teaching vector to our system.</p>
<p>It is another tool to achieve resiliency and to test scalability.</p>
<p>Chaos Engineering doesn’t bring down all your production system in an
unrecoverable way. It designs exercises that you and your team will use to
increase your operational experience and confidence.</p>
<p>Observability is a sort of requirement to understand how a chaotic event changes
the “normal” state of your system. But from another point of view a chaotic even
shed some light for a particular part of your system showing up lack of
monitoring and instrumentation.</p>
<p>There open source framework like <a href="https://github.com/chaostoolkit">Chaos-tookit</a>
or famous tools like <a href="https://github.com/Netflix/chaosmonkey">chaos-money</a>.</p>
<p>I will try to start with some very simple example without writing too much
code. I will get out from my system these metrics:</p>
<ol>
<li>Number of requests (probably from ingress/nginx)</li>
<li>The number of requests with status code &gt; 499</li>
<li>Http request latency</li>
</ol>
<p>After that, I will try to simulate an outage removing or scaling down particular
pods (the one that gets all the traffic) and I will look at how the metrics will
change and how long it takes to recover.</p>
OpenMetrics and the future of the prometheus exposition format2018-08-23T08:08:27+00:00http://gianarb.it/blog/prometheus-openmetrics-expositon-format<p>Who am I to tell you the future about the prometheus exposition format? Nobody!</p>
<p>I was at the PromCon in Munich in August 2018 and I found the conference great!
A lot of use cases about metrics, monitoring and prometheus itself. I work
at InfluxData and we was there as sponsor but I followed a lot of talks and I
had the chance to attend the developer summit the next day with a lot of
promehteus maintainers. Really good conversarsations!</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">I just
realized how lucky I was these days having chance to be so welcomed by the <a href="https://twitter.com/hashtag/prometheus?src=hash&amp;ref_src=twsrc%5Etfw">#prometheus</a>
community. I love my work. Thanks <a href="https://twitter.com/juliusvolz?ref_src=twsrc%5Etfw">@juliusvolz</a> <a href="https://twitter.com/TwitchiH?ref_src=twsrc%5Etfw">@TwitchiH</a> <a href="https://twitter.com/tom_wilkie?ref_src=twsrc%5Etfw">@tom_wilkie</a> and
everyone.. I feel regenerated</p>&mdash; :w !sudo tee % (@GianArb) <a href="https://twitter.com/GianArb/status/1028414240535793664?ref_src=twsrc%5Etfw">August
11, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>To be honest my scope few years ago was very different, I was working in PHP
writing webapplication that yes I was deploying but I wasn’t digging to much
around them and I was not smart enough to uderstand that all the pull vs push situation was just all garbage.
Smoke in the eyes that luckily I left behind me pretty soon because I had the
chance to meet smart people that drove me out.</p>
<p>Provide a comfortable way for me to expose and store metrics is a vital
request and the library needs to expose the RIGHT data it doesn’t matter if they
are pushing or pulling.</p>
<p>RIGHT means the best I can get to have more observability from an ops point of
view, but also from a business intelligence prospetive probably just
manipulating again the same data.</p>
<p>It is safe to say that a pull based exposition format is easy to pack together
because it works even if the server that should grab the exposed endpoint is
unavailable or even if nothing will grab them. A push based service will always
create some network noice even if nobody has interest on getting the metrics.</p>
<p>Back in the day we had SNMP but other than being an internet standard the
adoption is not comparable with the prometheus one, if we had how old it is and
how fast prometheus growed the situation gets even worst.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.1.0.0.0.1.1.0 octet_str "foo"
.1.0.0.0.1.1.1 octet_str "bar"
.1.0.0.0.1.102 octet_str "bad"
.1.0.0.0.1.2.0 integer 1
.1.0.0.0.1.2.1 integer 2
.1.0.0.0.1.3.0 octet_str "0.123"
.1.0.0.0.1.3.1 octet_str "0.456"
.1.0.0.0.1.3.2 octet_str "9.999"
.1.0.0.1.1 octet_str "baz"
.1.0.0.1.2 uinteger 54321
.1.0.0.1.3 uinteger 234
</code></pre></div></div>
<p>It also started as network exposing format, so it doesn’t express really well
other kind of metrics.</p>
<p>The <a href="https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md">prometheus exposition
format</a>
is extremly valuable and I recently instrumented a legacy application using the
prometheus sdk and my code looks a lot more clean and readable.</p>
<p>At the beginning I was using logs as transport layer for my metrics and time
series but I ended up having a lot of spam in log themself because I was also
streaming a lot of “not logs but metrics” garbage.</p>
<p>The link to the prometheus doc above is the best place to start, here I am just
copy pasting something form there:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code># HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000
# Escaping in label values:
msdos_file_access_time_seconds{path="C:\\DIR\\FILE.TXT",error="Cannot find file:\n\"FILE.TXT\""} 1.458255915e9
# Minimalistic line:
metric_without_timestamp_and_labels 12.47
# A weird metric from before the epoch:
something_weird{problem="division by zero"} +Inf -3982045
# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320
# Finally a summary, which has a complex representation, too:
# HELP rpc_duration_seconds A summary of the RPC duration in seconds.
# TYPE rpc_duration_seconds summary
rpc_duration_seconds{quantile="0.01"} 3102
rpc_duration_seconds{quantile="0.05"} 3272
rpc_duration_seconds{quantile="0.5"} 4773
rpc_duration_seconds{quantile="0.9"} 9001
rpc_duration_seconds{quantile="0.99"} 76656
rpc_duration_seconds_sum 1.7560473e+07
rpc_duration_seconds_count 2693
</code></pre></div></div>
<p>Think about that not as the prometheus way to grab metrics, but as the language
that your application uses to teach the outside world how does it feels.</p>
<p>It is just a plain text entrypoint over HTTP that everyone can parse and re-use.</p>
<p>For example
<a href="https://www.influxdata.com/time-series-platform/kapacitor/">kapacitor</a> or
<a href="https://www.influxdata.com/time-series-platform/telegraf/">telegraf</a> have
specific ways to parse and extract metrics from that URL.</p>
<p>If you don’t have time to write a parser for that you can use
<a href="https://github.com/prometheus/prom2json">prom2json</a> to get a JSON version of
that.</p>
<p>In Go you can dig a bit more inside that code and reuse some of functions for
example:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// FetchMetricFamilies retrieves metrics from the provided URL, decodes them</span><span class="x">
</span><span class="c">// into MetricFamily proto messages, and sends them to the provided channel. It</span><span class="x">
</span><span class="c">// returns after all MetricFamilies have been sent.</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">FetchMetricFamilies</span><span class="p">(</span><span class="x">
</span><span class="n">url</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="n">ch</span><span class="x"> </span><span class="k">chan</span><span class="o">&lt;-</span><span class="x"> </span><span class="o">*</span><span class="n">dto</span><span class="o">.</span><span class="n">MetricFamily</span><span class="p">,</span><span class="x">
</span><span class="n">certificate</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="n">key</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x">
</span><span class="n">skipServerCertCheck</span><span class="x"> </span><span class="kt">bool</span><span class="p">,</span><span class="x">
</span><span class="p">)</span><span class="x"> </span><span class="kt">error</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="nb">close</span><span class="p">(</span><span class="n">ch</span><span class="p">)</span><span class="x">
</span><span class="k">var</span><span class="x"> </span><span class="n">transport</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Transport</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">certificate</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="s">""</span><span class="x"> </span><span class="o">&amp;&amp;</span><span class="x"> </span><span class="n">key</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="s">""</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">cert</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">tls</span><span class="o">.</span><span class="n">LoadX509KeyPair</span><span class="p">(</span><span class="n">certificate</span><span class="p">,</span><span class="x"> </span><span class="n">key</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">tlsConfig</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">tls</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span><span class="x">
</span><span class="n">Certificates</span><span class="o">:</span><span class="x"> </span><span class="p">[]</span><span class="n">tls</span><span class="o">.</span><span class="n">Certificate</span><span class="p">{</span><span class="n">cert</span><span class="p">},</span><span class="x">
</span><span class="n">InsecureSkipVerify</span><span class="o">:</span><span class="x"> </span><span class="n">skipServerCertCheck</span><span class="p">,</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">tlsConfig</span><span class="o">.</span><span class="n">BuildNameToCertificate</span><span class="p">()</span><span class="x">
</span><span class="n">transport</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">http</span><span class="o">.</span><span class="n">Transport</span><span class="p">{</span><span class="n">TLSClientConfig</span><span class="o">:</span><span class="x"> </span><span class="n">tlsConfig</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x"> </span><span class="k">else</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">transport</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">http</span><span class="o">.</span><span class="n">Transport</span><span class="p">{</span><span class="x">
</span><span class="n">TLSClientConfig</span><span class="o">:</span><span class="x"> </span><span class="o">&amp;</span><span class="n">tls</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span><span class="n">InsecureSkipVerify</span><span class="o">:</span><span class="x"> </span><span class="n">skipServerCertCheck</span><span class="p">},</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">client</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="o">&amp;</span><span class="n">http</span><span class="o">.</span><span class="n">Client</span><span class="p">{</span><span class="n">Transport</span><span class="o">:</span><span class="x"> </span><span class="n">transport</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">decodeContent</span><span class="p">(</span><span class="n">client</span><span class="p">,</span><span class="x"> </span><span class="n">url</span><span class="p">,</span><span class="x"> </span><span class="n">ch</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p><a href="https://github.com/prometheus/prom2json/blob/master/prom2json.go#L123">FetchMetricsFamilies</a> can be used to get a channel with all the fetched
metrics. When you have the channel you can make what you desire:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">mfChan</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="nb">make</span><span class="p">(</span><span class="k">chan</span><span class="x"> </span><span class="o">*</span><span class="n">dto</span><span class="o">.</span><span class="n">MetricFamily</span><span class="p">,</span><span class="x"> </span><span class="m">1024</span><span class="p">)</span><span class="x">
</span><span class="k">go</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">prom2json</span><span class="o">.</span><span class="n">FetchMetricFamilies</span><span class="p">(</span><span class="n">flag</span><span class="o">.</span><span class="n">Args</span><span class="p">()[</span><span class="m">0</span><span class="p">],</span><span class="x"> </span><span class="n">mfChan</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">cert</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">key</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">skipServerCertCheck</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}()</span><span class="x">
</span><span class="n">result</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="o">*</span><span class="n">prom2json</span><span class="o">.</span><span class="n">Family</span><span class="p">{}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">mf</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">mfChan</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">result</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="nb">append</span><span class="p">(</span><span class="n">result</span><span class="p">,</span><span class="x"> </span><span class="n">prom2json</span><span class="o">.</span><span class="n">NewFamily</span><span class="p">(</span><span class="n">mf</span><span class="p">))</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>As you can see
<a href="https://github.com/prometheus/prom2json/blob/master/cmd/prom2json/main.go#L42"><code class="highlighter-rouge">prom2json</code></a>
converts the result to JSON.</p>
<p>It is pretty fleximple! And it is a common API to read applicatin status. A
common API we all know means automation! Dope automation!</p>
<h2 id="future">Future</h2>
<p>The prometheus exposition format growed in adoption across the board and a
couple of people leaded by <a href="https://twitter.com/TwitchiH">Richard</a> are now pushing
to have this format as new Internet Standard!</p>
<p>The project is called <a href="https://openmetrics.io/">OpenMetrics</a> and it is a Sandbox
project under CNCF.</p>
<p>if you are looking to follow the project here the official repository on
<a href="https://github.com/OpenObservability/OpenMetric">GitHub</a>.</p>
<p>Probably it looks just a political step with no value at all from a tech point of
view but I bet when it will be a standard and not just “the prometheus
exposition” we will start to have routers exposing stats over
<code class="highlighter-rouge">http://192.168.1.1/metrics</code> and it will be a lot of fun!</p>
<p>It will be obvious that it is not a <code class="highlighter-rouge">only-prometheus</code> feature and this new group
has people from difference companies and backgrounds. So the exposition format
will be probably not just for operational metrics but more generic.</p>
Apps I used during my nomad working2018-08-11T10:38:27+00:00http://gianarb.it/blog/nomad-working-apps<p>It is a couple of years now since my first conference and now that I am working
in remote, it is even harder to combine traveling and work.</p>
<p>Mainly because now that I don’t need to go to an office I travel more often, and
usually conference are a better more far away too. It means that I spent more
time, not in my usual workplace.</p>
<p>It is a very challenging and exciting opportunity, and I am glad to live it.
This is the first important things I think. If you are not happy about what you
do, even if it is challenging, you are going to give up.</p>
<p>But other things help me a lot. In general, they are come back to <code class="highlighter-rouge">planning.</code> I
feel better when I have the time to look around and to be prepared for the city
I am going to visit. There are a couple of apps that help me with that:</p>
<h2 id="workfrom"><a href="https://workfrom.co/">WorkFrom</a></h2>
<p>It is a community of digital nomad and remote worker. It keeps up to date a
database of pubs, libraries, restaurants, bars where you can work from. It is
very nice, and it has some nice feature like:</p>
<ol>
<li>A map, so you can see what is around you</li>
<li>Net speed measurement inside the app. Other than a detailed description of
the place it also shares (if the person who reviewed the area made it) how
the internet connection is fast and sometimes even the WiFi password, in this
way you don’t need to ask.</li>
<li>As I mentioned it is a community, so there is also a Slack Channel that you
can use to speak with another remote worker.</li>
</ol>
<p>I used it in Berlin, Munich, Copenaghen, Amsterdam and it worked pretty well.</p>
<h2 id="bike-and-other-local-transport-applications-i-am-using">bike and other local transport applications I am using</h2>
<p><a href="https://www.o.bike/it/">oBike</a> as an example because just because I used it
recently in Munich but I think you should always have a look to what the city
uses for bike sharing, for instance, because at least for me even if I love to
walk around take a ride time to time is faster and helpful.</p>
<p>Bonus point a lot of these apps have a free tier in means that you can even use
them for free the first time you visit that city!</p>
<h2 id="yelp">Yelp</h2>
<p>In general, I find <a href="https://www.yelp.com/">Yelp</a> better than TripAdvisor for
what concerns restaurant and place to eat. So when I am not able to spot nothing
good by myself during my walks around the city or when I am looking for a
specific kind of good I use Yelp.</p>
<h2 id="adobe-scan">Adobe Scan</h2>
<p>For this <a href="https://acrobat.adobe.com/us/en/mobile/scanner-app.html">app</a> I need to give
credit to <a href="https://twitter.com/fntlnz">Lorenzo</a> because he showed that app to me
the first time. I use it a lot after a trip when I need to submit the expenses.
It is always a very annoying work to do but at least with this app, I can take a
set of pictures, and it will generate a single pdf ready to be submitted!</p>
<h2 id="revolut">Revolut</h2>
<p>A few years ago when I was working at CurrencyFair I started to test
and play with online banks and <a href="https://revolut.com/r/gianlu1b2">Revolut</a> is
very good when you are traveling around, and you need to manage different
currencies. First of all, I like the idea to use a different card when I buy on
Amazon or when I travel because in case of any trouble I will have a limited
amount of money there. For example in Cube I had my card cloned but I only had
10 Euro, so it was not the first count. (the bank give me the money back in any
case btw).</p>
<p>Plus Revolut has some excellent features to track where and what you spent. You
can label your transfers to easy lookup them when you need to expense or
calculate how much you spent. The exchange commission low compared with the more
traditional banks, so this is a free win!</p>
<p>Let me know on <a href="https://twitter.com/gianarb">Twitter</a> if you have any other
applications to suggest I will be happy to try them next time and maybe to add
them here!</p>
<p><small><a href="https://www.newhdwallpapers.in/natural-hd-wallpapers/himalayas-mountain-series-tibet/" target="_blank">hero img credits</a> </small></p>
FAQ: Distributed tracing2018-07-06T10:38:27+00:00http://gianarb.it/blog/faq-distributed-tracing<p>This article is a write up of a talk that I will give at the
<a href="https://osmc.de">osmc</a> in Germany in November about Distributed tracing.
It is a sequence of questions I got about distributed systems and distributed
monitoring.</p>
<h2 id="why-do-i-need-distributed-tracing">Why do I need distributed tracing?</h2>
<p>It always depends, I find distributed tracing useful in a microservices
environment or more in general when there is a request that flies in a system
crossing different applications, queues or processes.
If you have a problem understanding where a request fails, you need to
<em>follow it</em> in some way and tracing does just that.</p>
<h2 id="how-do-you-follow-a-request">how do you follow a request?</h2>
<p>First of all, we should probably change the name <em>request</em>, it looks too HTTP
oriented, and it is not really what we look for now. In modern application, you
are interested in <em>events</em>. You need to monitor an event:</p>
<ul>
<li>user registration</li>
<li>payment</li>
<li>a bank transaction</li>
<li>send an email</li>
<li>generate an invoice</li>
</ul>
<p>These are all events, and probably in your system, they are distributed not via
HTTP but maybe they go in a queue, or they are broadcasted using Kafka or Redis.
Distributed tracing is all about tracking events. The way to go is to create an
id. Usually, it is called <code class="highlighter-rouge">request_id</code> or <code class="highlighter-rouge">trace_id</code> and you need a way to
propagate it in your system.</p>
<p>For example, in a queue, you can put the <code class="highlighter-rouge">trace_id</code> as part of the payload. Via
HTTP or gRPC you can use Headers.</p>
<p>Your application can take that id, and it can create the span to trace a
particular section.</p>
<h2 id="how-a-trace-looks-like">how a trace looks like?</h2>
<p><img src="/img/trace.jpg" alt="How I image a trace for a distributed tracing app" class="img-fluid" /></p>
<p>In my mind, this is a picture for one trace. Every segment is a span.
So, every span has a trace id, and every span has its own <code class="highlighter-rouge">span_id</code>.
You can attach information to every span as key value store. Let’s suppose a
span represent a query in mysql you can put the query as metadata in the span
itslef. In this way you will have a bit more context.</p>
<h2 id="do-we-need-a-standard-for-tracing">do we need a standard for tracing?</h2>
<p>I can’t convince you that interoperability is essential if you already analyzed
the problem and you answered “No” to yourself.
To build a trace you need to agree on something over languages and
applications.
That’s why I think a standard is something you can not avoid, at the end you
will end up having one just for your company.</p>
<h2 id="how-a-tracing-infrastructure-looks">how a tracing infrastructure looks?</h2>
<p><img src="/img/tracing_infra.png" style="width:70%" alt="Sketch of tracing infrastructure." class="img-fluid" /></p>
<p>The applications that are writing traces is not important. Traces is cross
platform and languages. Usually, you
point an app to a tracer. It can be Zipkin, Jaeger or others.</p>
<p>The tracer takes all the traces, and it stores them in a storage. The databases are
usually ElasticSearch, Cassandra, InfluxDB. It depends on which tracer you are
using. They support different databases.</p>
<p>In general traces are high cardinality oriented data, and you can write a lot
of them in a short amount of time. So it is a write-intensive
application.</p>
<p>There are a couple of other pieces that you can add in your tracing
infrastructure:</p>
<ul>
<li>You can add a <em>downsampler</em> to select what to store. If an API request generate
too many traces probably you are interested in storing only a % of them to
decrease pressure on your database. So you can use a simple distributed hash
algorithm on the trace_id to declare what to save or not. A <code class="highlighter-rouge">mod</code> on the
<code class="highlighter-rouge">trace_id</code> is enough for example.</li>
<li>You can add a <em>collector</em> in front of the tracer. Zipkin support Kafka For
example. In InfluxDB we use telegraf. A collector is usually a stateless
application, it gets all the traces from the applications. It bulks them and
sends them to the tracer. A collector decreases the pressure on the tracer
itself because usually, they work better with a bulk of data. In second if a
tracer go down or you need to update it, the collector is a layer that can
keep the traces for a little bit to give you time to restore the tracer.</li>
</ul>
<h2 id="why-did-i-pick-opentracing">why did I pick opentracing?</h2>
<p>I am an interoperability oriented developer; I think it is essential to avoid
vendor lock-in and embracing a big community like the opentracing one you get
a lot of tools and services already instrumented with this protocol. It makes
my life easy.</p>
<h2 id="can-i-have-a-tracing-infrastructure-on-prem">can I have a tracing infrastructure on-prem?</h2>
<p>You can; there are a couple of tracers open source.</p>
<ul>
<li><a href="https://zipkin.io/">Zipkin</a> is an open source
project in Java started by Twitter.</li>
<li><a href="https://github.com/jaegertracing/jaeger">Jaeger</a> looks a lot like a porting of
it in Golang and Uber makes it.</li>
</ul>
<p>Both of them are open source, and they support different backends like
ElasticSaerch, Cassandra and so on.</p>
<h2 id="there-are-as-a-service-tracing-infrastructure">there are as a service tracing infrastructure?</h2>
<p>There are, NewRelic has an opentracing compatible API, or
<a href="https://lightstep.com/">Lightstep</a> for example. A lot of cloud providers offer
a tracing service. AWS X-RAY or Google Stackdriver.</p>
<h2 id="can-i-store-traces-everywhere">can I store traces everywhere?</h2>
<p>You can, but they are a high cardinality data. The <code class="highlighter-rouge">trace_id</code> is usually the
lookup parameter for your queries. It means that it should be indexed, but it
changes for every request. The consequence is a big index.
You need to keep it in mind.</p>
<h2 id="once-you-do-the-tracethen-what">Once you do the trace…then what?</h2>
<p>I left this question as the last one because I read it in the opentracing
mailing list and I think it is a hilarious question.</p>
<p>First of all, you don’t buy a pan and after the fact you start asking yourself
why you have it.</p>
<p>Probably you need to write something, and for that reason, you
buy a pen.</p>
<p>Anyway, I trace my applications because it helps me to understand my environment
over the “distribution complexity.” I can detect what is taking too long and a
trace helps me to understand what to optimize.</p>
<p>Opentracing has a set of standard annotation very useful to detect network
latency between services. You can mark a span as “client send” request for
example. And when the server gets the request, you can mark another span as
“server received.” This two information is useful to know how much time your
request spends going from client to the server and you can optimize them time
usually working on the proximity between these two applications.</p>
<p>More in general you can parse a trace to get what ever you need as normal logs
or events the powerful things is downsampling and analysis.
If you are tracing a queue system you can get the average time for a worker to
process a message.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Let me know if you have more questions on twitter
<a href="https://twitter.com/gianarb">@gianarb</a>. I am happy to answer them here.</p>
Logs, metrics and traces are equally useless2018-06-18T10:38:27+00:00http://gianarb.it/blog/logs-metrics-traces-aggregation<p>Every signal from applications or infrastructure is useless, in the distributed
system era aggregation matters.</p>
<p>The ability to combine logs, metrics and, traces together is the key takeaway
here.</p>
<p>Kubernetes spin ups too many containers to allow us to stream or tail a log
fail.</p>
<p>Even cloud providers offer too many virtual machines to enable us to tail
logs.</p>
<p>A centralized place where to store all of them is a great start, but you
need to experience and learn how to combine the metrics you are ingesting to
increase the visibility over your system.</p>
<p>If you instrument your code with
opentracing, for example, you can get the <code class="highlighter-rouge">trace_id</code> and attach it to your log
to associate it with the trace itself. It can also work as the lookup key for
troubleshooting.</p>
<p>If you get some weird logs, you will know from where it comes.
With opentracing, this is still a bit of a mess the specification recently
<a href="https://github.com/opentracing/specification/blob/master/rfc/trace_identifiers.md">added explicit support to extract TraceId and SpanId from the
SpanContext</a>.
It is currently not implemented in a lot of implementation. I recently started a
conversation in the
<a href="https://github.com/opentracing/opentracing-go/issues/188">opentracing-go</a>
project to figure out how to apply it because currently it depends from what
tracer you are using and it is an essential regression for the specification
itself that should hide it by design.</p>
<p>Using Jaeger this is the way to do it:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if sc, ok := span.Context().(jaeger.SpanContext); ok {
sc.TraceID()
}
</code></pre></div></div>
<p>Using Zipkin:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>zipkinSpan, ok := sp.Context().(zipkin.SpanContext)
if ok == true &amp;&amp; zipkinSpan.TraceID.Empty() == false {
w.Header().Add("X-Trace-ID", zipkinSpan.TraceID.ToHex())
}
</code></pre></div></div>
<p>To get back in track, I wrote this article because I saw this problem and this
inclination speaking with friends, colleagues and other devs, we are now good
(or just better) storing high cardinality values but save them inside a database
doesn’t give us any value it is all about how we use them.</p>
<p>Correlation brings your alert to a different level. You probably have an alarm
to measure how much disks you still have.</p>
<p>An alert on the only CPU usage can be very frustrated
even more if it happens too often and a lot of time you restart a container or a
node to make it work because at 2 am you can’t fix the cause. You can
investigate what matters to fill an issue on GitHub.</p>
<p>Every automation tools can make your work leaving you free to sleep. It can
probably fill out the issue.</p>
<p>Combining the CPU with the time for the system to recover from a node restart
can make your alert smart enough to wake you up when it is not able to fix
itself leaving you ready for more acute and trivial problems.</p>
<h2 id="conclusion">Conclusion</h2>
<p>It is a pretty straightforward concept, but yes, everything is useless if you
store data without getting values out of them doesn’t matter if they are logs,
metrics or traces. The real value is not in a single one of them, it is in how
do you aggregate them together because a complex simple doesn’t explain itself
over one signal.</p>
Cloud Native Intranet with Kubernetes, CoreDNS and OpenVPN2018-05-29T10:38:27+00:00http://gianarb.it/blog/cloud-native-intranet<p>This article has a marketing and buzzword oriented title. I know.</p>
<p>Let me introduce you to what I am going to speak with you here with better
worlds: VPN, private, DNS, kubernetes, security.</p>
<p>I hope we all agree that VPN should be a must-have when you set up an
infrastructure. It doesn’t matter what you are doing, how many people are
working with you.</p>
<p>When you design a new system usually you need to expose to the public only some
service over HTTP and HTTPS all the rest: Jenkins, monitoring tools,
dashboards, log management should be locked-down and accessible just in a
private network. An intranet.</p>
<blockquote>
<p>An intranet is a private network accessible only to an organization’s staff.
Often, a wide range of information and services are available on an
organization’s internal intranet that is unavailable to the public, unlike the
Internet.</p>
</blockquote>
<p>All these concepts apply to “Cloud Native” ecosystem as well.</p>
<p>Kubernetes has a powerful dashboard and CTL that you can use to interact with
the API. That API doesn’t need to be publicly exposed, and to use the CLI from
your laptop, you should set up a VPN.</p>
<h2 id="openvpn">OpenVPN</h2>
<p>Usually, I configure an OpenVPN using the image
<a href="https://hub.docker.com/r/kylemanna/openvpn/">kylemanna/openvpn</a> available on
Docker Hub. It is straightforward to apply, and it offers a set of utilities
around user creation and certification management.</p>
<div class="language-yml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Service</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">nodePort</span><span class="pi">:</span> <span class="s">1194</span>
<span class="na">port</span><span class="pi">:</span> <span class="s">1194</span>
<span class="na">protocol</span><span class="pi">:</span> <span class="s">UDP</span>
<span class="na">targetPort</span><span class="pi">:</span> <span class="s">1194</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">NodePort</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">apps/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Deployment</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">namespace</span><span class="pi">:</span> <span class="s2">"</span><span class="s">openvpn"</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">replicas</span><span class="pi">:</span> <span class="s">1</span>
<span class="na">strategy</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">Recreate</span>
<span class="na">selector</span><span class="pi">:</span>
<span class="na">matchLabels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">template</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">labels</span><span class="pi">:</span>
<span class="na">app</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">spec</span><span class="pi">:</span>
<span class="na">nodeSelector</span><span class="pi">:</span>
<span class="na">role</span><span class="pi">:</span> <span class="s">vpn</span>
<span class="na">containers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">docker.io/kylemanna/openvpn</span>
<span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">/etc/openvpn/setup/configure.sh"</span><span class="pi">]</span>
<span class="na">env</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VPN_HOSTNAME</span>
<span class="na">valueFrom</span><span class="pi">:</span>
<span class="na">configMapKeyRef</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">vpn-hostname</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">hostname</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">VPN_DNS</span>
<span class="na">valueFrom</span><span class="pi">:</span>
<span class="na">configMapKeyRef</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">vpn-hostname</span>
<span class="na">key</span><span class="pi">:</span> <span class="s">dns</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">containerPort</span><span class="pi">:</span> <span class="s">1194</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">securityContext</span><span class="pi">:</span>
<span class="na">capabilities</span><span class="pi">:</span>
<span class="na">add</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">NET_ADMIN</span>
<span class="na">volumeMounts</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/etc/openvpn/setup</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span>
<span class="pi">-</span> <span class="na">mountPath</span><span class="pi">:</span> <span class="s">/etc/openvpn/certs</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">certs</span>
<span class="na">readOnly</span><span class="pi">:</span> <span class="no">false</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">configMap</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">openvpn</span>
<span class="na">defaultMode</span><span class="pi">:</span> <span class="s">0755</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">certs</span>
<span class="na">persistentVolumeClaim</span><span class="pi">:</span>
<span class="na">claimName</span><span class="pi">:</span> <span class="s">openvpncerts</span>
</code></pre></div></div>
<p>I put the persistentVolumeClaim to remember you to store in a persisted and
safe place (and you should backup them too) the certificates used and generated
by the VPN <code class="highlighter-rouge">/etc/openvpn/certs</code>.</p>
<p>I won’t write more about this topic; we are all excellent yaml developers!</p>
<p>How to create users, configuration and so on is a well knows topic that you can
easily <a href="https://openvpn.net/index.php/open-source/documentation.html">find in the OpenVPN’s
documentation</a>.</p>
<p>I don’t know if you realized that, but this VPN runs inside a Kubernetes
Cluster, so well configurated allow us to reach pods via a private network and
a bonus point also via kubedns to ping services, pods and all the other
resources registered to it.</p>
<p>To do that OpenVPN server can be configured to push kubedns to the client:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dhcp-option DNS &lt;kube-dns-ip&gt;
</code></pre></div></div>
<p>Something with learned is that if you are using Linux the
NetworkManager-OpenVPN plugin pushes the DNS correctly, but the OpenVPN cli
tool doesn’t if you are using the last one you need to set it up in another
way.</p>
<p>Tips: You can take the <code class="highlighter-rouge">&lt;kube-dns-ip&gt;</code> doing <code class="highlighter-rouge">cat /etc/resolv.conf</code> from inside a pod.</p>
<h2 id="dns">DNS</h2>
<p>Push the KubeDNS or the DNS used by kubernetes is not enough to have a complete
intranet. You should be able to set up a custom domain to have friendly or
short URL.</p>
<p>You can take two different directions. KubeDNS can have static
record configured, but some person is not happy to touch or customize too much
the KubeDNS because Kubernetes itself use it and if you mess it up all it can
be a problem.</p>
<p>A possible solution is to deploy another DNS like CoreDNS and
configures it to resolve KubeDNS as a fallback. In this way, you will be free
to register custom LTDs and records. Kubernetes is going to use KubeDNS as
usual, and if you mess up CoreDNS, only a fraction of your system will blow
out.</p>
<p>Naturally to resolve your custom domains from the VPN you need to push
the CoreDNS ip and not the one used by Kubernetes.</p>
<p>If two DNSs are too much take the option one or from Kubernetes 1.10 you can
use CoreDNS as kubernetes DNS so it is a bit more flexible and you can use only
that one if you are brave enough.</p>
<p>I suggested CoreDNS because it supports records configuration via
<a href="https://github.com/coredns/coredns/tree/master/plugin/etcd">etcd</a>. Here an
example of Corefile:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>. {
errors
etcd *.myinternal {
stubzones
path /skydns
entrypoint http://etcd-1:2379,http://etcd-2:2379,http://etcd-3:2379
upstream /etc/resolv.conf
}
proxy . /etc/resolv.conf
}
</code></pre></div></div>
<p>Running this configuration inside a pod automatically fallback to kubedns (that
automatically fallback to the one configured to reach internet). Because of
<code class="highlighter-rouge">upstream</code> point to <code class="highlighter-rouge">resolv.conf</code> that inside a pod contains kubedns.</p>
<h2 id="benefits">Benefits</h2>
<p>Resolve Kubernetes DNS record from your local environment is very comfortable
to build a shared or dynamic development environment for you and your
colleagues.</p>
<p>You can set up per-developer namespaces that they can use to
deploy services reachable from the program that they are writing. Or you can
deploy your application, and another person connected to the VPN will be able
to use it.</p>
Server time vs Response time2018-05-18T10:38:27+00:00http://gianarb.it/blog/server-time-vs-response-time<p>If you find yourself in San Francisco walking nearby Market Street, you should
consider stopping at the Jewish Museum. There is a charming Pastrami place just
next to it. It is a sandwich place with good lemonade. It only takes 3-4 minutes
to get your meal, and from there it takes no more than 15 minutes walking to be
in front of the Ocean. Very nice!
Now, let’s consider this other scenario.
It is lunchtime, and you are starving. You rush outside your office, and you run
to the Pastrami place close to the Jewish Museum. After 35 minutes of wait, you
get your sandwich and start eating it asking yourself: why it took so long this
time? Shall I probably have walked to the next place to get a faster meal?</p>
<p>Something similar can happen to your Services as well! And that’s precisely the
phenomena in computer science we try to capture using the concepts of server
time and response time.
Server time aims to measures how much a server takes to run a specific action.
Let’s say consider an example operation the generation of a monthly report: it
usually takes 2ms, but if a lot of customers require the same kind of report at
the same time and your system saturates? This situation might very quickly end
up in having a subset of them getting the report in more than 1 minute or
actually in the timeout of the operation. The time it takes for a customer to
get his report is what is typically called response time.</p>
<h2 id="how-can-we-measure-these-metrics">How can we measure these metrics?</h2>
<p>The answer to this question is not easy: it depends on your architecture and
system. The starting point is instrumenting your application to determine how
much time it gets to produce the report. Stress testing is the other important
aspect: generating some load on your application and sampling the average
response time will let you estimate the application’s service time. Notice that
to make this measurement the app should NOT soak during this test!</p>
<p>If you control all the chain (from the HTTP app that sends the request to the
server), you can trace the request and simulate the same behavior of your
customers. If you can’t do this, you can consider using the frontend edge,
probably a load balancer.</p>
<blockquote>
<p>I would rather have questions that can’t be answered than answers that can’t
be questioned. Richard Feynman</p>
</blockquote>
<h2 id="why-does-it-matter">Why does it matter</h2>
<p>How many nodes do I need to deploy to accommodate x number of requests per
second? When should I consider scaling out my application? How does scale-out
affect the customer experience?
This is precisely why server time and response time matters! Having an average
response time close to the defined service time is a signal of proper
utilization and health of an application because it indicates that the response
latency is under control and it is far from saturation. Bringing to the limit
these two signals, in addition, is a key metric to estimate the correct sizing
of the applications instances and infrastructure.</p>
<p><img alt="Market Street San Francisco, Pastrami Resturant Jewish Meseum" src="/img/pastrami-sf.jpg" class="img-fluid" /></p>
<p>Btw the Pastrami place exists! You should try it! I will be in SF in 2 weeks. So
let me know about other places <a href="https://twitter.com/gianarb">@gianarb</a>.
Picture from GMaps. I will take a better one!</p>
Go how to cleanup HTTP request terminated.2018-04-25T10:38:27+00:00http://gianarb.it/blog/go-http-cleanup-http-connection-terminated<p>Expensive HTTP handler is everywhere, doesn’t matter how good you are as a
developer. Business logic is what matters in our application, and it can be
pretty complicated. It can create large files, resources on AWS starts
thousands of containers on Kubernetes.</p>
<p>This kind of procedures have in common they can be very slow and they produce a
lot of garbage if the system/person who requires that stops prematurely by
mistake or not.</p>
<p>If your API requests create AWS resources and the client, terminate the call
you should clean what you created.</p>
<p>if you are generating a report and the customer changes are mind and refresh
you should stop the procedure.</p>
<p>You bet! Queues, background processes probably fit better but coming back on
the previous example, if you are computing something and who is waiting for the
result changed his mind, stop and release resources can be a massive
optimization.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>package main
import <span class="o">(</span>
<span class="s2">"fmt"</span>
<span class="s2">"io/ioutil"</span>
<span class="s2">"net/http"</span>
<span class="s2">"os"</span>
<span class="s2">"time"</span>
<span class="o">)</span>
func main<span class="o">()</span> <span class="o">{</span>
http.HandleFunc<span class="o">(</span><span class="s2">"/a"</span>, func<span class="o">(</span>w http.ResponseWriter, r <span class="k">*</span>http.Request<span class="o">)</span> <span class="o">{</span>
err :<span class="o">=</span> ioutil.WriteFile<span class="o">(</span>os.TempDir<span class="o">()</span>+<span class="s2">"/txt"</span>, <span class="o">[]</span>byte<span class="o">(</span><span class="s2">"hello"</span><span class="o">)</span>, 0644<span class="o">)</span>
<span class="k">if </span>err <span class="o">!=</span> nil <span class="o">{</span>
panic<span class="o">(</span>err<span class="o">)</span>
<span class="o">}</span>
println<span class="o">(</span><span class="s2">"new file "</span> + os.TempDir<span class="o">()</span> + <span class="s2">"/txt"</span><span class="o">)</span>
notify :<span class="o">=</span> w.<span class="o">(</span>http.CloseNotifier<span class="o">)</span>.CloseNotify<span class="o">()</span>
go func<span class="o">()</span> <span class="o">{</span>
&lt;<span class="nt">-notify</span>
println<span class="o">(</span><span class="s2">"The client closed the connection prematurely. Cleaning up."</span><span class="o">)</span>
os.Remove<span class="o">(</span>os.TempDir<span class="o">()</span> + <span class="s2">"/txt"</span><span class="o">)</span>
<span class="o">}()</span>
time.Sleep<span class="o">(</span>4 <span class="k">*</span> time.Second<span class="o">)</span>
fmt.Fprintln<span class="o">(</span>w, <span class="s2">"File persisted."</span><span class="o">)</span>
<span class="o">})</span>
http.ListenAndServe<span class="o">(</span><span class="s2">":8080"</span>, nil<span class="o">)</span>
<span class="o">}</span>
</code></pre></div></div>
<p>When you are building an HTTP server in Go, you can use a channel provided by
the Zhttp.ResponseWriter` to wait for the connection to be closed. And if it
happens, you can take action. The prototype above is very simple, every
request stores a file but I would like, remove the file if the client closes
the connection.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>run main.go
</code></pre></div></div>
<p>You can start the server, and from another terminal, you can start a <code class="highlighter-rouge">curl</code>, you
will see that after almost 4 seconds your request will succeed and the file
will be persisted on disk. Check it!</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ time curl http://localhost:8080/a
File persisted.
real 0m4.018s
user 0m0.008s
sys 0m0.006s
$ cat /tmp/txt
</code></pre></div></div>
<p>Now let’s suppose that the client terminates the connection because it is too
slow or the person who made the request doesn’t care anymore.
Are you going to leave that request going? Event if nobody cares and it is just
consuming resources?</p>
<p>As you can see I am using the Notifier to remove the file if the client
terminates the connection:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">notify</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">w</span><span class="o">.</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">CloseNotifier</span><span class="p">)</span><span class="o">.</span><span class="n">CloseNotify</span><span class="p">()</span><span class="x">
</span><span class="k">go</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="o">&lt;-</span><span class="n">notify</span><span class="x">
</span><span class="nb">println</span><span class="p">(</span><span class="s">"The client closed the connection prematurely. Cleaning up."</span><span class="p">)</span><span class="x">
</span><span class="n">os</span><span class="o">.</span><span class="n">Remove</span><span class="p">(</span><span class="n">os</span><span class="o">.</span><span class="n">TempDir</span><span class="p">()</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="s">"/txt"</span><span class="p">)</span><span class="x">
</span><span class="p">}()</span><span class="x">
</span></code></pre></div></div>
<p>You can check it stopping a <code class="highlighter-rouge">curl</code> just after starting it:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ time curl http://localhost:8080/a
^C
real 0m1.016s
user 0m0.008s
sys 0m0.005s
</code></pre></div></div>
<p>And the server reports</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ go run main.go
new file /tmp/txt
The client closed the connection prematurely. Cleaning up.
</code></pre></div></div>
<p>That’s it! Build and clean after yourself!</p>
Go testing tricks2018-04-17T10:38:27+00:00http://gianarb.it/blog/go-testing-tricks<p>I recently wrote a blog post with my <a href="/blog/testing-shit">point of view about
testing</a>. I used Go as the language to concretize it. I had
good feedback about that article, and this is all about how I write tests in
Go.</p>
<h2 id="fixtures">Fixtures</h2>
<p>I wrote that I don’t like them, but I think they are useful. You can use them
to verify the same function checking the same assertion with different input.
So let’s say you are testing a function that returns the multiplication of two
numbers if the first is even, it returns the division if not.</p>
<p>I will write two tests, one to test events number and one to test the other
case, and I will set up two fixtures one for every test. I won’t write just one
test with elaborate fixtures because they are hard to read and the name of the
test function will help a lot to understand the assertion. Small example good
for blogging purpose. But I hope you got the idea.</p>
<div class="language-golang highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span><span class="x"> </span><span class="n">test</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="s">"testing"</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">MagicFunction</span><span class="p">(</span><span class="n">f</span><span class="x"> </span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="x"> </span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="kt">int</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">f</span><span class="o">%</span><span class="m">2</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="m">0</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">f</span><span class="x"> </span><span class="o">*</span><span class="x"> </span><span class="n">s</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">f</span><span class="x"> </span><span class="o">/</span><span class="x"> </span><span class="n">s</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">TestEventInputsShouldReturnMoltiplication</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">table</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">first</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">second</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">result</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="p">}{</span><span class="x">
</span><span class="p">{</span><span class="m">2</span><span class="p">,</span><span class="x"> </span><span class="m">1</span><span class="p">,</span><span class="x"> </span><span class="m">2</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="m">4</span><span class="p">,</span><span class="x"> </span><span class="m">10</span><span class="p">,</span><span class="x"> </span><span class="m">40</span><span class="p">},</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">table</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">MagicFunction</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">first</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">second</span><span class="p">);</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">result</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"Got %d, expected %d. They should be the same."</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">result</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">TestOddInputsShouldReturnDivision</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">table</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">first</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">second</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">result</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="p">}{</span><span class="x">
</span><span class="p">{</span><span class="m">15</span><span class="p">,</span><span class="x"> </span><span class="m">3</span><span class="p">,</span><span class="x"> </span><span class="m">5</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="m">21</span><span class="p">,</span><span class="x"> </span><span class="m">7</span><span class="p">,</span><span class="x"> </span><span class="m">3</span><span class="p">},</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">table</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">MagicFunction</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">first</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">second</span><span class="p">);</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">result</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"Got %d, expected %d. They should be the same."</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">result</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<h2 id="sub-test">sub-test</h2>
<p>To make the fixtures a bit better I use the <code class="highlighter-rouge">t.Run</code> function a lot. It is a
feature introduced in Go 1.9 as part of the <code class="highlighter-rouge">testing</code> package.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span><span class="x"> </span><span class="n">test</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"fmt"</span><span class="x">
</span><span class="s">"testing"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">MagicFunction</span><span class="p">(</span><span class="n">f</span><span class="x"> </span><span class="kt">int</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="x"> </span><span class="kt">int</span><span class="p">)</span><span class="x"> </span><span class="kt">int</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">f</span><span class="o">%</span><span class="m">2</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="m">0</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">f</span><span class="x"> </span><span class="o">*</span><span class="x"> </span><span class="n">s</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">f</span><span class="x"> </span><span class="o">/</span><span class="x"> </span><span class="n">s</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">TestEventInputsShouldReturnMoltiplication</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">table</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">first</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">second</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="n">result</span><span class="x"> </span><span class="kt">int</span><span class="x">
</span><span class="p">}{</span><span class="x">
</span><span class="p">{</span><span class="m">2</span><span class="p">,</span><span class="x"> </span><span class="m">1</span><span class="p">,</span><span class="x"> </span><span class="m">2</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="m">4</span><span class="p">,</span><span class="x"> </span><span class="m">10</span><span class="p">,</span><span class="x"> </span><span class="m">40</span><span class="p">},</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="n">_</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="k">range</span><span class="x"> </span><span class="n">table</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Run</span><span class="p">(</span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"%d * %d"</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">first</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">second</span><span class="p">),</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">MagicFunction</span><span class="p">(</span><span class="n">s</span><span class="o">.</span><span class="n">first</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">second</span><span class="p">);</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">result</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">t</span><span class="o">.</span><span class="n">Errorf</span><span class="p">(</span><span class="s">"Got %d, expected %d. They should be the same."</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">,</span><span class="x"> </span><span class="n">s</span><span class="o">.</span><span class="n">result</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">})</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p><code class="highlighter-rouge">vim-go</code> has an option <code class="highlighter-rouge">let g:go_test_show_name=1</code> to allow the name of the
test as part of the output for :GoTest or :GoTestFunc. This helps a lot to
enjoy this feature.</p>
<h2 id="golden-files">Golden files</h2>
<p>Golden files are something used in different packages in the Go standard
library, and Michael Hashimoto spoke about it during his brilliant talk about
testing at the <a href="https://www.youtube.com/watch?v=8hQG7QlcLBk">GopherCon 2017</a>.
In case of complex output, you can verify the result of the tests with the
content of a file. It improves order and readability. When you declare a
global flag in your test, it becomes available inside <code class="highlighter-rouge">go test</code> so if you use
the update flags all the tests will pass, but you will update all the golden
files. So this is very useful if you need to compare a lot of bytes.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">update</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">flag</span><span class="o">.</span><span class="n">Bool</span><span class="p">(</span><span class="s">"update-golden-files"</span><span class="p">,</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x"> </span><span class="s">"Update golden files."</span><span class="p">)</span><span class="x">
</span></code></pre></div></div>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>go <span class="nb">test</span> <span class="nt">-update-golden-files</span>
</code></pre></div></div>
<p>I was using this trick a lot when I was writing PHP code, and I was testing HTTP responses.</p>
<h2 id="test-helper-and-return-function">Test helper and return function</h2>
<p>When you have repeatable code across tests, you can create a helper function,
and you can use it in your tests. There are two general rules about this
approach:</p>
<ol>
<li>The helper function should have access to *t testing.T variable.</li>
<li>Your helper never returns an error; it marks the test as failed. That’s why
it needs access to <code class="highlighter-rouge">*t. testing.T</code>.</li>
</ol>
<p>Another good trick is to return a function from the helper to clean up what you
did in the helper. So let’s say that your helper starts an HTTP server. You can
return the HTTP Close function as a callback.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="p">()</span><span class="x"> </span><span class="n">testHelperStartHTTPServer</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">ts</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">httptest</span><span class="o">.</span><span class="n">NewServer</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">HandlerFunc</span><span class="p">(</span><span class="k">func</span><span class="p">(</span><span class="n">w</span><span class="x"> </span><span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="c">// long and complex mock maybe with a golden file and so on</span><span class="x">
</span><span class="p">}))</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="k">func</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x"> </span><span class="n">ts</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x"> </span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">TestYourTest</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">hclose</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">testhelperStartHTTPServer</span><span class="p">()</span><span class="x">
</span><span class="c">// All your logic and checks</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="n">hclose</span><span class="p">()</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>I used the same practice when I was writing integration tests using bash and
<a href="https://github.com/sstephenson/bats">bats</a>. It is a very clean and easy to
read approach.</p>
<h2 id="parallel">parallel</h2>
<p>You can use the function <code class="highlighter-rouge">t.Parallel()</code> to notify at the test runner that your
case can run in parallel with other tests marked as parallel. When you write
unit tests, you can almost always run them in parallel because they should be
completely isolated.</p>
<h2 id="short-and-verbose">Short and verbose</h2>
<p><code class="highlighter-rouge">-short</code> and <code class="highlighter-rouge">-v</code> are two flags available when you run <code class="highlighter-rouge">go test</code>. You can use
them in your tests:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import "testing"
func TestVeryLongAndExpensiveCapability(t *testing.T) {
if testing.Short() {
t.Skip("skipping this testsVeryLongAndExpensive is too expensive")
}
// ... other code
}
</code></pre></div></div>
<p><code class="highlighter-rouge">-short</code> describes itself pretty well, you can skip tests that are too long and expensive.</p>
<p><code class="highlighter-rouge">-v</code> allows you to print more:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>import "testing"
func TestVeryLongAndExpensiveCapability(t *testing.T) {
if testing.Verbose() {
}
// ... other code
}
</code></pre></div></div>
<h2 id="testingquick">testing/quick</h2>
<p><a href="https://golang.org/pkg/testing/quick/">testing/quick</a> is a nice package that
offers a set of utilities to write test quick. Go has not an assertion library
inside the stdlib but this can help if you are like me and you are happy to not
vendor assertion libraries because <code class="highlighter-rouge">if { }</code> with some sugar is what I need.</p>
<p>So that’s it, have fun and write tests!</p>
The Go awesomeness2018-04-09T10:38:27+00:00http://gianarb.it/blog/go-awesomeness<p>It’s one year since I started to use Go every day at work. I was using it before
but for fun or OSS projects. I was looking for my next challenge, and I was
mainly working with PHP, JavaScript previously and I knew that a compiled,
statically typed language was my next step.</p>
<p>At my previous job at CurrencyFair, the environment was pretty standard for a
financial tech company so backend in Java, frontend in PHP. But my experience
with all the interfaces and abstract classes that I created in Java at that time
made me hate that language. So I was looking for something different.</p>
<p>I was as I am now involved in automation, cloud and
operational other than development so all the tools like Docker, InfluxDB,
Kubernetes, Consul, Vault was in golang and for me as OSS addicted it become the
natural choice.
Now after all this time I am ready to write why I think Go is the right choice
for me now.</p>
<h2 id="1-abstraction-and-maintainability">1. abstraction and maintainability</h2>
<p>I wrote a lot about <a href="/blog/the-abstract-manifesto">this topic</a>, so I am not going to repeat myself. But I
think maintainability is tied together with abstraction. Previously when I was
working with PHP, we always had services, injection and so on. In that
environment it was good, but all that abstraction like in Java doesn’t make your
code more flexible. It makes it hard to understand in the long run and code
needs to be written with history in mind because delete code is very hard.
Go with its interface implementation, how it forces you to struct the project
helps the codebase to grow in a better way.</p>
<h2 id="2-stdlib">2. Stdlib</h2>
<p>Community wasted time across languages to identify the right way to indent code.
Go comes with that decision done. Same for testing. How to write automatic
tests, benchmarks is inside the language. No libraries, it is there.
More in general os, net, net.http, img and so on, a lot of stuff are provided by
the language itself. It is great because you don’t need anything to start,
other than Go. Compared with other languages you can do a lot more things.
Having all this feature inside Go guarantees compatibility over time, they won’t
break compatibility for the next years, and the code is developed and reviewed
by a large number of people.</p>
<h2 id="3-pprof">3. pprof</h2>
<p>pprof is a profiler, and it is shipped as part of Go. You can use it even via
cli, or it also has an excellent HTTP package under
<a href="https://golang.org/pkg/net/http/pprof/">net/HTTP/pprof</a>.
Just to show you how much power it can be InfluxDB extends it to export a zip
archive with all the information we need to troubleshoot the database behavior:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="p">(</span><span class="n">h</span><span class="x"> </span><span class="o">*</span><span class="n">Handler</span><span class="p">)</span><span class="x"> </span><span class="n">handleProfiles</span><span class="p">(</span><span class="n">w</span><span class="x"> </span><span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">switch</span><span class="x"> </span><span class="n">r</span><span class="o">.</span><span class="n">URL</span><span class="o">.</span><span class="n">Path</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">case</span><span class="x"> </span><span class="s">"/debug/pprof/cmdline"</span><span class="o">:</span><span class="x">
</span><span class="n">httppprof</span><span class="o">.</span><span class="n">Cmdline</span><span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">)</span><span class="x">
</span><span class="k">case</span><span class="x"> </span><span class="s">"/debug/pprof/profile"</span><span class="o">:</span><span class="x">
</span><span class="n">httppprof</span><span class="o">.</span><span class="n">Profile</span><span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">)</span><span class="x">
</span><span class="k">case</span><span class="x"> </span><span class="s">"/debug/pprof/symbol"</span><span class="o">:</span><span class="x">
</span><span class="n">httppprof</span><span class="o">.</span><span class="n">Symbol</span><span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">)</span><span class="x">
</span><span class="k">case</span><span class="x"> </span><span class="s">"/debug/pprof/all"</span><span class="o">:</span><span class="x">
</span><span class="n">h</span><span class="o">.</span><span class="n">archiveProfilesAndQueries</span><span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">)</span><span class="x">
</span><span class="k">default</span><span class="o">:</span><span class="x">
</span><span class="n">httppprof</span><span class="o">.</span><span class="n">Index</span><span class="p">(</span><span class="n">w</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>Here all the code
<a href="https://github.com/influxdata/influxdb/blob/442581d299b7d642e073bbe42112fa9b58fb071a/services/httpd/pprof.go#L21">influxdata/influxdb</a>.
This is super useful because we can ask customers or developer in the OSS
community to export and upload the archive to see what is going on.
Having a standard way to troubleshoot and export a profile allows us to build
visualization or static analysis on it for common calculation.</p>
<h2 id="4-delve">4. delve</h2>
<p>A good debugging session is the best way to approach a new application or to go
deeper learning a language or a software.
<a href="https://github.com/derekparker/delve">delve</a> is easy to setup and to use. Even
if you are not gdb/debugger superhero as I am not you will be able to make your
first steps with delve. So it is a nice starting point too.</p>
<h2 id="5-godoc">5. godoc</h2>
<p>Other than behind an excellent way to generate documentation from source code I
use it a lot even when I am not designed libraries just to double check that my
package has the comprehensive public methods. I always think about what I am
exposing to the outside when I write code. APIs are not just JSON or HTTP thing,
every object exposes their API, and you need to be aware of how you are building
iteration between the internal state and the outside. Avoid misuse of your
structs is your responsibility as developer and godoc help me to identify poor
decision.</p>
<h2 id="6-vim-go">6. vim-go</h2>
<p>I would like to stay in my terminal all day, and vim-go allows me to write good
code in my comfort zone. In the past I wrote a lot of vim scripts and plugins,
following how fatih and all the other maintainers are developing
<a href="https://github.com/fatih/vim-go">vim-go</a> is great.
Bonus point they recently added support for delve, so you can now debug golang
application in vim!</p>
<h2 id="7-dep">7. dep</h2>
<p>Dependency management is probably the worst things that Go has. The good thing
is that now we have [dep] and it should become the standard way to manage
dependencies. Right now the situation looks a lot like this:</p>
<p><img class="img-fluid" src="/img/fight-club.jpg" /></p>
<p>Govender, go get, glide currently there are a lot of different ways to manage
dependencies, and it generates a lot of confusion, but I hope at the end we will
converge in just one. Probably dep.</p>
<h2 id="conclusion">Conclusion</h2>
<p>More in general with Go I am learning that the language is one of expect to
become a good developer. A good developer needs to know the language, but the
best way to go deeper in it is writing tests, benchmarks, profiling application
and using the debugger. All these tools make my life as developer easy. Easy
life for me means that I can go deeper solving problems and indirectly it will
make me a better developer.</p>
<p>Go is fun!</p>
Observability according to me2018-04-04T10:38:27+00:00http://gianarb.it/blog/observability<p>I started to read about observability almost one year ago, <a href="https://twitter.com/mipsytipsy">Charity
Major</a>
comes to my mind when I thinking about this topic and she is the person that is
pushing very hard on this.</p>
<p>This is probably the natural evolution of how we approach monitoring.</p>
<p>Distributed systems require a different way to approach the three monitoring
piles: collect, storage and analytics.</p>
<p>Understand a microservices environment brings a new layer of complexity and the
most obvious consequence is the amount of data that we are storing compared, I
think it is way more than before.</p>
<p>Prometheus, InfluxDB, TimescaleDB, Cassandra and all the time series databases
that show up every week is a clear sign that now we need more than just a way to
store metrics. I think is now clear that collecting more metrics is the point.
More data is not directly related to a deeper understanding of our system.</p>
<p>Observability for a lot of companies look like a new way to sell analytics
platform but according to me it’s a scream to bring us back to the problem: “How
can we understand what is happening?” or even better “How should we use the data
we have to understand what’s going on?”.</p>
<p>All the data should be organized, reliable and usable. Logs, metrics, traces
are part of the resolution, the brain to analyze and get value out from them is
what Observability means to me.</p>
<p>Visualization is one expect, proactive monitoring, correlation, and hierarchy
are other steps. Looking at our old graphs all of them are driven by the
hostname for example. But now we have containers, we have virtual machines and
immutable infrastructure makes rebuilding less costly and more secure than an
incremental update. The name of the server should not be the keyword for our
queries, the focus should be moved to the role of services.</p>
<p>Think about your Kubernetes cluster, you label servers based on what they will
run, if something unusual happens the first things to do is to move the node out
from the production pool, the autoscaler will replace it and you will be
troubleshoot it later.</p>
<p>Before we were looking at processes, we were keeping them alive as the Olympic
flame but containers are making them volatile. We spin them up and down for
every request in some serverless environment. What we care and what we should
monitor are the events that float across our services, that’s the new gold. W we
can lose 1000 containers but we can’t miss the purchase order made my a
customer. All our effort should be moved on that side.</p>
<p>I love this point of view because it brings us to what really matters, our
applications.</p>
<p><img src="/img/mountain-garbage.jpg" class="img-fluid" /></p>
<p>According to me the mountain of waste showed in the picture explains really well
our current situation, we collected what ended up to be a lot of garbage and now
we need to climb it looking for a better point of view. I think the data in our
time series databases are not garbage but gold, it’s just not simple as it
should be.</p>
<p>That’s why is great that companies are building tools to fill the gap:
<a href="https://github.com/influxdata/ifql">IFQL</a> is an example. The idea behind the
project is to build a language to query and manipulate data in an easy way. Same
for company line Honeycomb or open source projects like Grafana and Chronograf
that are trying to make these data easy to use.</p>
<p>We spoke about tools but there is another big expect and it’s all a cultural,
distributed teams need different tools to collaborate and troubleshoot problems.
Different UI and way to interact with graph and metrics.</p>
I don't give a shit about testing2018-03-29T10:38:27+00:00http://gianarb.it/blog/testing-shit<p>That’s what I learned during my experience as a developer. Doesn’t matter
which languages you end up working if you are making HTTP API or things like
that you don’t have an excuse. Write tests make your development faster, and it will
drastically improve maintainability of your project.</p>
<p>In this post, I would like to tell you how I approach testing particularly in
Go obviously.</p>
<p>First of all, when you create a new file, you should write its <code class="highlighter-rouge">_test.go</code> child.
It’s hard to tell you who should be the child of who. Sometimes I start
to write everything inside a test function, just because run the actual test is faster
compared with compile, run the binary, trigger the right entry point and so on.
When I am satisfied, I move the
code to a function, and I leave the assertions I wrote as a new test. <strong>Pretty good</strong>.</p>
<blockquote>
<p>I don’t give a shit about automated testing. I write tests.</p>
</blockquote>
<p>I use <a href="https://github.com/fatih/vim-go"><code class="highlighter-rouge">vim-go</code></a> and <code class="highlighter-rouge">:GoTestFunc</code> is probably
the most used shortcut during my day to day job.</p>
<p>When I can choose I don’t use assertion libraries, the <code class="highlighter-rouge">testing</code> package is
enough for me and dependency management in go is a pain, so fewer things I vendor
better I feel about myself.</p>
<p><img src="/img/laziness.jpg" class="img-fluid" /></p>
<p>I use fixtures, but I don’t like them. I prefer to write more small tests than complicated fixtures.</p>
<p>A single test for me is more descriptive, and I don’t mind to write redundant code, I can always refactor it later or move it some helper function. A complicated fixture will be hard to maintain.
The name of the function is an excellent way to describe what you are covering in your test and the function itself
creates a beautiful block that improves readability.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">TestCarComposition</span><span class="p">(</span><span class="n">t</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">fixtures</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="p">[]</span><span class="n">car</span><span class="o">.</span><span class="n">Composition</span><span class="p">{</span><span class="x">
</span><span class="p">{</span><span class="s">"blue"</span><span class="p">,</span><span class="x"> </span><span class="s">"europe"</span><span class="p">,</span><span class="x"> </span><span class="m">1</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">,</span><span class="x"> </span><span class="s">"2011-12-05"</span><span class="p">,</span><span class="x"> </span><span class="s">"ford"</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="m">35</span><span class="p">,</span><span class="x"> </span><span class="kt">bool</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">,</span><span class="x"> </span><span class="s">"ford"</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="s">"red"</span><span class="p">,</span><span class="x"> </span><span class="s">"usa"</span><span class="p">,</span><span class="x"> </span><span class="m">0</span><span class="p">,</span><span class="x"> </span><span class="kt">bool</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">,</span><span class="x"> </span><span class="s">"fiat"</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="s">"white"</span><span class="p">,</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="m">35</span><span class="p">,</span><span class="x"> </span><span class="kt">bool</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">,</span><span class="x"> </span><span class="s">"kia"</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="s">"orange"</span><span class="p">,</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="m">1</span><span class="p">,</span><span class="x"> </span><span class="kt">bool</span><span class="p">,</span><span class="x"> </span><span class="s">"2010-05-12"</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">},</span><span class="x">
</span><span class="p">{</span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="s">""</span><span class="p">,</span><span class="x"> </span><span class="m">0</span><span class="p">,</span><span class="x"> </span><span class="kt">bool</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">,</span><span class="x"> </span><span class="s">"ford"</span><span class="p">},</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>Bonus point, as you can see fixtures are sad to read!</p>
<p>Even unit vs. integration vs. function is a very annoying discussion. Don’t tell me
about TDD, BDD, CCC, DDD things. I don’t care they are all amazing as
soon as they can make my development simple.</p>
<p>So, CDD is probably my best test methodology: <strong>Comfort driven development</strong>.</p>
<p>Usually when I am writing a computing function when it elaborates maps, strings,
files without using too many external resources I start from unit test, because
it makes iteration faster as I told before. And it won’t require too many mocks.
I don’t like mocks.</p>
<h2 id="lets-discuss-mocks">Let’s discuss mocks</h2>
<p>Mocks are a pain; you end up to be bored when you write mocks, they won’t fail when it’s useful for you to see an error and they will fail when you don’t care.
So comfort looks very far from mocks!</p>
<p>When mocks becomes too complicated, and I can write another kind of tests I go with that solution. Maybe integration or I will try to write the simple mock
possible, sometimes even the entire web server can be a valuable solution:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">TestInfluxDBSdkGetTheRightValues</span><span class="p">(</span><span class="n">ti</span><span class="x"> </span><span class="o">*</span><span class="n">testing</span><span class="o">.</span><span class="n">T</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">ts</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">httptest</span><span class="o">.</span><span class="n">NewServer</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">HandlerFunc</span><span class="p">(</span><span class="k">func</span><span class="p">(</span><span class="n">w</span><span class="x"> </span><span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">data</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">influxdb</span><span class="o">.</span><span class="n">Response</span><span class="p">{</span><span class="x">
</span><span class="n">Results</span><span class="o">:</span><span class="x"> </span><span class="p">[]</span><span class="n">influxdb</span><span class="o">.</span><span class="n">Result</span><span class="p">{</span><span class="x">
</span><span class="p">{</span><span class="x">
</span><span class="n">Series</span><span class="o">:</span><span class="x"> </span><span class="p">[]</span><span class="n">influxdbModels</span><span class="o">.</span><span class="n">Row</span><span class="p">{},</span><span class="x">
</span><span class="p">},</span><span class="x">
</span><span class="p">},</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">Header</span><span class="p">()</span><span class="o">.</span><span class="n">Add</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span><span class="x"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">WriteHeader</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">StatusOK</span><span class="p">)</span><span class="x">
</span><span class="n">_</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">json</span><span class="o">.</span><span class="n">NewEncoder</span><span class="p">(</span><span class="n">w</span><span class="p">)</span><span class="o">.</span><span class="n">Encode</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="x">
</span><span class="p">}))</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="n">ts</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x">
</span><span class="n">config</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">influxdb</span><span class="o">.</span><span class="n">HTTPConfig</span><span class="p">{</span><span class="n">Addr</span><span class="o">:</span><span class="x"> </span><span class="n">ts</span><span class="o">.</span><span class="n">URL</span><span class="p">}</span><span class="x">
</span><span class="n">client</span><span class="p">,</span><span class="x"> </span><span class="n">_</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">influxdb</span><span class="o">.</span><span class="n">NewHTTPClient</span><span class="p">(</span><span class="n">config</span><span class="p">)</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="n">client</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x">
</span><span class="c">// Whatever you need to check</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>You need to play carefully; these tests are slower and more expensive in
resources.
But I like the idea to take the faster solution when I am developing; you can
come back on your tests later when the feature is more stable and better
designed. Write tests should not slow me down too much, I am looking for a way
to write the implementation and the test fasts to iterate on both of them other than waste time making everything perfect. Nothing will be forever; nothing will be complete in programming, so design your environment to be easy to change.</p>
<h2 id="integration-tests">Integration tests</h2>
<p>I am a CLI kind of person, so I often send HTTP requests via cURL.
Docker is very easy from day one to start and stop your application,
cleaning databases and so on.</p>
<p><a href="https://github.com/sstephenson/bats"><code class="highlighter-rouge">bats</code></a> combines these two sentences. It is an automation test framework for
bash. It is very simple to setup, and it allows me to copy paste some cURL, and
with jq, you can make the checks you need over your JSON response.</p>
<p>An integration test suite made with bats looks like that:</p>
<ol>
<li>An “init” file in bash where you can run setup and teardown function before and after every test. Usually, you can you that functions to spin up and down the
containers that you need to tests, this is the one that I wrote for this
example</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="k">function </span>setup<span class="o">()</span> <span class="o">{</span>
<span class="nv">teardownCallback</span><span class="o">=</span><span class="k">$(</span>init<span class="k">)</span>
<span class="o">}</span>
<span class="k">function </span>teardown<span class="o">()</span> <span class="o">{</span>
<span class="nb">eval</span> <span class="nv">$teardownCallback</span>
<span class="o">}</span>
<span class="k">function </span>getHost<span class="o">()</span> <span class="o">{</span>
<span class="nb">echo</span> <span class="s2">"http://localhost"</span>
<span class="o">}</span>
<span class="k">function </span>init <span class="o">{</span>
<span class="nv">executionID</span><span class="o">=</span><span class="k">$(</span><span class="nb">cat</span> /dev/urandom | tr <span class="nt">-dc</span> <span class="s1">'a-zA-Z0-9'</span> | fold <span class="nt">-w</span> 7 | head <span class="nt">-n</span> 1<span class="k">)</span>
<span class="nv">containerLabels</span><span class="o">=</span><span class="s2">"exec=</span><span class="k">${</span><span class="nv">executionID</span><span class="k">}</span><span class="s2">"</span>
<span class="k">$(</span>docker run <span class="nt">-d</span> <span class="nt">-l</span> <span class="nv">$containerLabels</span> <span class="nt">-p</span> 80:80 nginx<span class="k">)</span>
<span class="nb">echo</span> <span class="s2">"docker ps -aq -f 'label=</span><span class="k">${</span><span class="nv">containerLabels</span><span class="k">}</span><span class="s2">' | xargs docker rm -f"</span>
<span class="o">}</span>
</code></pre></div></div>
<ol>
<li>You have a set of <code class="highlighter-rouge">.bats</code> files with the various scenarios, I wrote one to
check if the status code 200 for the nginx home</li>
</ol>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env bats</span>
load utils
@test <span class="s2">"Nginx home return 200"</span> <span class="o">{</span>
<span class="nv">statusCode</span><span class="o">=</span><span class="k">$(</span>curl <span class="nt">-I</span> <span class="nt">-X</span> GET <span class="s2">"</span><span class="k">$(</span>getHost<span class="k">)</span><span class="s2">"</span> 2&gt;/dev/null | head <span class="nt">-n</span> 1 | cut
<span class="nt">-d</span><span class="s1">$' '</span> <span class="nt">-f2</span><span class="k">)</span>
<span class="o">[</span> <span class="nv">$statusCode</span> <span class="nt">-eq</span> 200 <span class="o">]</span>
<span class="o">}</span>
</code></pre></div></div>
<p>What you are running is a <code class="highlighter-rouge">bats</code> test to check that <code class="highlighter-rouge">nginx:latest</code> is serving the right page.
Your use case will be ten times more complicated.</p>
<p>Another reason to take this approach is about bash itself. If you are not a bash
expert, you will probably end up to write straightforward tests, cURL, grep,
regex and some pipes. Nothing more.</p>
<p>And you won’t use any code that runs your application. It’s important to avoid
weird buggy tests.</p>
<h2 id="developer-happiness">developer happiness</h2>
<p>Tests are a methodology to decrease the cost of maintenance and to improve your
ability to write code.</p>
<p>It should not be a fashion way to show how good you are as a developer. You will
be a good developer as a side effect.</p>
<p>I look at all the different way to tests my code as a tool set, AI is becoming
very smart. So we need to be less “server” and more human been. 100% coverage
for unit tests looks a lot like something that a server can do. Pick the right
method based on your feeling.</p>
<script>
$(document).ready(function() {
$('body').css("background", "#F5F3E6");
});
</script>
Review book Database Reliability Engineer2018-03-27T09:08:27+00:00http://gianarb.it/blog/database-reliability-engineer-review<p>The author of <a href="http://amzn.to/2FE5z4V">Database Reliability Engineer</a> are Laine Campbell, Charity Majors.
It is published by O’Reilly. You can buy it on Amazon.</p>
<p>You probably know the book <a href="https://gianarb.it/blog/site-reliability-engineering-review">Site Reliability
Engineer</a>, if you don’t I reviewed
it a few days ago.</p>
<p><img src="/img/dbre-book.jpg" class="img-fluid" /></p>
<p>This book walks the same experience but focused on databases. I work for
InfluxData as SRE and I deal every days with databases running on our Cloud
Product so I am in that topic but now because I am good as DBA but just as a
developer with a background on distributed system and cloud computing.</p>
<p>It doesn’t really matter if you manage databases as DBA or if you use it as a
developer this book contains useful contents for both categories.</p>
<p>It is a dance book, I started it ready it few months ago and at some point I
spotted, just because it contains too many notions and experiences and it
requires some time to metabolize them.</p>
<p>I particularly enjoyed the Chapter 7 about <strong>Backup and Recovery</strong>. But also Chapter
3 about about <strong>Risk Management</strong> is great because it goes deeper on how metrics
should drive your way to look at risks and outage.</p>
<p>My daily job is all around database orchestration, containers and so on. So I
found this book very useful for my day to day job and it enforced the expectation
that I had about the application that I am building.</p>
<p>I will try to go deeper on backup management to make the recovery part of a more
structure pipeline to be sure that it is always usable for example.</p>
<p>It is a very practical book based and you can fill all the enthusiasm and the
experiences made by the two authors Charity and Laine. If you are happy to
learn from who has its hands dirty this book is your book. It drives you an
story and less learned.</p>
<p>How to manage migration and how to fill the gap between developer and DBA
because they are both the same goals the success of the company and the project.</p>
<p>If we can keep both on the same loop avoiding backstabbing we will increase our
chance of success. If you are a manages and you see some friction in your teams,
this book can give you some good feedback.</p>
How to use a Forwarding Proxy with golang2018-03-21T09:38:27+00:00http://gianarb.it/blog/golang-forwarding-proxy<p>A forwarding proxy is a proxy configuration that handle requests from a set of
internal clients that are trying to create a connection to the outside.</p>
<p>In practice is a man in the middle between your application and the server that you are
trying to connect. It works over the HTTP(S) protocol and it is implemented at the
edge of your infrastructure.</p>
<p>Usually, you can find it in large organizations or universities and it is used as
additional control mechanism for authorization and security.</p>
<p>I find it useful when you work with containers or in a dynamic cloud environment
because you will have a set of servers for all the outbound network
communication.</p>
<p>If you work in a dynamic environment as AWS, Azure and so on you will end up
having a variable number of servers and also a dynamic number of public IPs.
Same if your application runs on a Kubernetes cluster. Your container can be
everywhere.</p>
<p>Now let’s suppose that a customer asks you to provide a range of public IPs
because he needs to set up a firewall… How can
you provide this feature? In some environments can be very simple, in others
very complicated.</p>
<p>1st December 2015 a users asked this question on the <a href="https://discuss.circleci.com/t/circleci-source-ip/1202">CircleCI
forum</a> this request is
still open. This is just an example, CircleCi is great. I am not complaining
about them.</p>
<p>One of the possible ways to fix this problem is via the forwarding proxy. You can
spin up a set of nodes with a static ip and you can offer the list to the
customer.</p>
<p>Almost all cloud providers have a way to do that, floating ip
on DigitalOcean or Elastic IP on AWS.</p>
<p>You can configure your applications to forward the requests to that pool and
the end services will get the ip from the forward proxy nodes and not from the
internal one.</p>
<p>This can be another security layer for your infrastructure because you will be
able to control and scan packages that are going outside from your network in a
really simple way and in a centralized place.</p>
<p>This is not a single point of failure because you can spin up more than one
forward proxies and they scale really well.</p>
<p>Under the hood, a forward proxy is the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT">HTTP method
<code class="highlighter-rouge">CONNECT</code></a>.</p>
<blockquote>
<p>The CONNECT method converts the request connection to a transparent TCP/IP
tunnel, usually to facilitate SSL-encrypted communication (HTTPS) through an
unencrypted HTTP proxy.</p>
</blockquote>
<p>A lot of HTTP Client across languages already support this in a very
transparent way. I built a very small example using golang and
<a href="https://www.privoxy.org/">privoxy</a> to show you how simple it is.</p>
<p>First of all, let’s build an application called <code class="highlighter-rouge">whoyare</code>. It is an HTTP server
that returns your remote address:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span><span class="x"> </span><span class="n">main</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"encoding/json"</span><span class="x">
</span><span class="s">"net/http"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">http</span><span class="o">.</span><span class="n">HandleFunc</span><span class="p">(</span><span class="s">"/whoyare"</span><span class="p">,</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">w</span><span class="x"> </span><span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">Header</span><span class="p">()</span><span class="o">.</span><span class="n">Set</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span><span class="x"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="x">
</span><span class="n">body</span><span class="p">,</span><span class="x"> </span><span class="n">_</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">json</span><span class="o">.</span><span class="n">Marshal</span><span class="p">(</span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span><span class="x">
</span><span class="s">"addr"</span><span class="o">:</span><span class="x"> </span><span class="n">r</span><span class="o">.</span><span class="n">RemoteAddr</span><span class="p">,</span><span class="x">
</span><span class="p">})</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">body</span><span class="p">)</span><span class="x">
</span><span class="p">})</span><span class="x">
</span><span class="n">http</span><span class="o">.</span><span class="n">ListenAndServe</span><span class="p">(</span><span class="s">":8080"</span><span class="p">,</span><span class="x"> </span><span class="no">nil</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>You can call the <code class="highlighter-rouge">GET</code> the route <code class="highlighter-rouge">/whoyare</code> and you will receive a JSON like
<code class="highlighter-rouge">{"addr": "34.35.23.54"}</code> where <code class="highlighter-rouge">34.35.23.54</code> is your public address. Running
<code class="highlighter-rouge">whoyare</code> from your laptop if you make a request on your terminal you should get
<code class="highlighter-rouge">localhost</code> as remote address. You can use curl to try it:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>18:36 <span class="nv">$ </span>curl <span class="nt">-v</span> http://localhost:8080/whoyare
<span class="k">*</span> TCP_NODELAY <span class="nb">set</span>
<span class="o">&gt;</span> GET /whoyare HTTP/1.1
<span class="o">&gt;</span> User-Agent: curl/7.58.0
<span class="o">&gt;</span> Accept: <span class="k">*</span>/<span class="k">*</span>
<span class="o">&gt;</span>
&lt; HTTP/1.1 200 OK
&lt; Content-Type: application/json
&lt; Date: Sun, 18 Mar 2018 17:36:40 GMT
&lt; Content-Length: 31
&lt;
<span class="k">*</span> Connection <span class="c">#0 to host localhost left intact</span>
<span class="o">{</span><span class="s2">"addr"</span>:<span class="s2">"localhost:38606"</span><span class="o">}</span>
</code></pre></div></div>
<p>I wrote another application, it uses <code class="highlighter-rouge">http.Client</code> to print the response on
stdout. If you have the server running you can run it:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span><span class="x"> </span><span class="n">main</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"io/ioutil"</span><span class="x">
</span><span class="s">"log"</span><span class="x">
</span><span class="s">"net/http"</span><span class="x">
</span><span class="s">"os"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">type</span><span class="x"> </span><span class="n">whoiam</span><span class="x"> </span><span class="k">struct</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">Addr</span><span class="x"> </span><span class="kt">string</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">()</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">url</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="s">"http://localhost:8080"</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="s">""</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Getenv</span><span class="p">(</span><span class="s">"URL"</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">url</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">os</span><span class="o">.</span><span class="n">Getenv</span><span class="p">(</span><span class="s">"URL"</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"Target %s."</span><span class="p">,</span><span class="x"> </span><span class="n">url</span><span class="p">)</span><span class="x">
</span><span class="n">resp</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">http</span><span class="o">.</span><span class="n">Get</span><span class="p">(</span><span class="n">url</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="s">"/whoyare"</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="n">resp</span><span class="o">.</span><span class="n">Body</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x">
</span><span class="n">body</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">ioutil</span><span class="o">.</span><span class="n">ReadAll</span><span class="p">(</span><span class="n">resp</span><span class="o">.</span><span class="n">Body</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">())</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="nb">println</span><span class="p">(</span><span class="s">"You are "</span><span class="x"> </span><span class="o">+</span><span class="x"> </span><span class="kt">string</span><span class="p">(</span><span class="n">body</span><span class="p">))</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>So, this is a very simple example, but you can apply this example to more
complex environments.</p>
<p>To make this example a bit more clear I created two virtual machines on
DigitalOcean, one will run privoxy the other one will run <code class="highlighter-rouge">whoyare</code>.</p>
<ul>
<li><strong>whoyare</strong>: public ip 188.166.17.88</li>
<li><strong>privoxy</strong>: public ip 167.99.41.79</li>
</ul>
<p>Privoxy is a very simple to setup forward proxy, nginx, haproxy doesn’t fit very
well for this use case because they do not support the CONNECT method.</p>
<p>I built a docker image
<a href="https://hub.docker.com/r/gianarb/privoxy/"><code class="highlighter-rouge">gianarb/privoxy</code></a>, it’s on Docker
Hub. You can run and by default, it runs on port 8118.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>core@coreos-s-1vcpu-1gb-ams3-01 ~ <span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="nt">--rm</span> <span class="nt">-p</span> 8118:8118
gianarb/privoxy:latest
2018-03-18 17:28:05.589 7fbbf41dab88 Info: Privoxy version 3.0.26
2018-03-18 17:28:05.589 7fbbf41dab88 Info: Program name: privoxy
2018-03-18 17:28:05.591 7fbbf41dab88 Info: Loading filter file:
/etc/privoxy/default.filter
2018-03-18 17:28:05.599 7fbbf41dab88 Info: Loading filter file:
/etc/privoxy/user.filter
2018-03-18 17:28:05.599 7fbbf41dab88 Info: Loading actions file:
/etc/privoxy/match-all.action
2018-03-18 17:28:05.600 7fbbf41dab88 Info: Loading actions file:
/etc/privoxy/default.action
2018-03-18 17:28:05.607 7fbbf41dab88 Info: Loading actions file:
/etc/privoxy/user.action
2018-03-18 17:28:05.611 7fbbf41dab88 Info: Listening on port 8118 on IP address
0.0.0.0
</code></pre></div></div>
<p>The second step is to build and scp <code class="highlighter-rouge">whoyare</code> in your server. You can
build it using the command:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ CGO_ENABLED=0 GOOS=linux go build -o bin/server_linux -a ./whoyare
</code></pre></div></div>
<p>Now that we have the application up and running we can try via cURL to query it
directly and via privoxy.</p>
<p>Let’s try directly as we did previously:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl -v http://your-ip:8080/whoyare
</code></pre></div></div>
<p><code class="highlighter-rouge">cURL</code> uses an environment variable <code class="highlighter-rouge">http_proxy</code> to forward the requests through
the proxy:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ http_proxy=http://167.99.41.79:8118 curl -v http://188.166.17.88:8080/whoyare
* Trying 167.99.41.79...
* TCP_NODELAY set
* Connected to 167.99.41.79 (167.99.41.79) port 8118 (#0)
&gt; GET http://188.166.17.88:8080/whoyare HTTP/1.1
&gt; Host: 188.166.17.88:8080
&gt; User-Agent: curl/7.58.0
&gt; Accept: */*
&gt; Proxy-Connection: Keep-Alive
&gt;
&lt; HTTP/1.1 200 OK
&lt; Content-Type: application/json
&lt; Date: Sun, 18 Mar 2018 17:37:02 GMT
&lt; Content-Length: 29
&lt; Proxy-Connection: keep-alive
&lt;
* Connection #0 to host 167.99.41.79 left intact
{"addr":"167.99.41.79:58920"}
</code></pre></div></div>
<p>As you can see I have set <code class="highlighter-rouge">http_proxy=http://167.99.41.79:8118</code> and the response
doesn’t contain my public ip but the proxy one.</p>
<p><img src="/img/frankenstain-jr.jpg" alt="" /></p>
<p>These are the logs that you should expect from privoxy for the requests crossing it:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>2018-03-18 17:28:22.886 7fbbf41d5ae8 Request: 188.166.17.88:8080/whoyare
2018-03-18 17:32:29.495 7fbbf41d5ae8 Request: 188.166.17.88:8080/whoyare
</code></pre></div></div>
<p>The client that you run it previously by default it connects to <code class="highlighter-rouge">localhost:8080</code>
but you can override the target URL via env var <code class="highlighter-rouge">URL=http://188.166.17.88:8080</code>.
Running the following command I reached directly <code class="highlighter-rouge">whoyare</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ URL=http://188.166.17.88:8080 ./bin/client_linux
2018/03/18 18:37:59 Target http://188.166.17.88:8080.
You are {"addr":"95.248.202.252:38620"}
</code></pre></div></div>
<p>The golang <code class="highlighter-rouge">HTTP.Client</code> supports a set of environment
variables to configure the proxy, it makes everything very flexible because
passing
these variables to any service already running it will just work.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export HTTP_PROXY=http://http_proxy:port/
export HTTPS_PROXY=http://https_proxy:port/
export NO_PROXY=127.0.0.1, localhost
</code></pre></div></div>
<p>The first two are very simple, one is the proxy for the HTTP requests, the
second for HTTPS. <code class="highlighter-rouge">NO_PROXY</code> excludes a set of hostname, the hostname listed
there won’t cross the proxy. In my case localhost and 127.0.0.1.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HTT_PROXY=http://forwardproxy:8118
+--------------+ +----------------+ +----------------+
| | | | | |
| client +----------^+ forward proxy +--------^+ whoyare |
| | | | | |
+--------------+ +----------------+ +----^-----------+
|
|
+---------------+ |
| | |
| client +-------------------------------------------+
| |
+---------------+
HTTP_PROXY not configured
</code></pre></div></div>
<p>The client with the environment variables configured will cross the forward
proxy. Other client will reach it directly.</p>
<p>This granularity is very important. It’s very flexible because other than a
“per-process” you can also select what request to forward and what to exclude.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ HTTP_PROXY=http://167.99.41.79:8118 URL=http://188.166.17.88:8080
./bin/client_linux
2018/03/18 18:39:18 Target http://188.166.17.88:8080.
You are {"addr":"167.99.41.79:58922"}
</code></pre></div></div>
<p>As you can see we just reached <code class="highlighter-rouge">whoyare</code> via proxy and the <code class="highlighter-rouge">addr</code> in response is
now ours but the proxy one.</p>
<p>The last command is a bit weird but it is just to show how the <code class="highlighter-rouge">NO_PROXY</code> works.
We are calling the proxy excluding the <code class="highlighter-rouge">whoyare</code> URL, and as expected it doesn’t
cross the proxy:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ HTTP_PROXY=http://167.99.41.79:8118 URL=http://188.166.17.88:8080 NO_PROXY=188.166.17.88 ./bin/client_linux
2018/03/18 18:42:03 Target http://188.166.17.88:8080.
You are {"addr":"95.248.202.252:38712"}
</code></pre></div></div>
<p>Let’s read this article as a practical introduction to golang, forward proxy. You can
subscribe to my <a href="/atom.xml">rss feed</a> or you can follow me on
<a href="https://twitter.com/gianarb">@twitter</a>. Probably I will write about how to
replace <code class="highlighter-rouge">privoxy</code> with golang and about how to setup and deploy this solution on
Kubernetes. So let me know what to write first!</p>
The abstract manifesto2018-03-17T10:08:27+00:00http://gianarb.it/blog/the-abstract-manifesto<p>This is a personal outburst. Stop to abstract by default.</p>
<p>I worked on too many application abstracted by default. And abstracted I mean
complicated.</p>
<p>The abstraction is easy to understand when you need it. If you need to think too
much about why that crappy code should have an interface, or that made should
implement an interface and not an object you are out of track.</p>
<p>Abstraction is not the answer, code architecture is, unit testing helps,
integration tests are the key to the modern microservices environment.</p>
<p>Don’t waste your time creating interfaces that nothing will reuse. If you don’t
know what to do run.</p>
<p>There are languages and design pattern that probably set your brain to look for
abstraction everywhere. I worked with Java developer that wasn’t able to write a
class without an interface, or without its abstract. My question was: “Why are
we doing that?”. Compliance.</p>
<blockquote>
<p>Dude, your world is a very boring one, and you are the root cause.</p>
</blockquote>
<p>If you are working in a service-oriented environment with services big enough
to be rewritten easily the abstraction is even more useless.</p>
<p>We are developers, we often don’t build rockets. That’s life, there are a good
amount of companies that make rockets, apply there or you will put your company
in the condition of paying technical debts for you and they will hire smart
contractors to figure out what you did now that you are not working there
because after probably just one year you locked yourself in that boring project
full of complicated concepts.</p>
<p>btw, I don’t think that software to control rockets has a lot of abstractions
sorry.</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Y&#39;all are all about passionate programmers, but honestly I&#39;d
rather programmers than care _just enough_. I could do with less pedantic
arguments about code.</p>&mdash; ｡ 𝕷𝖎𝖓𝖉𝖘𝖊𝖞 𝕭𝖎𝖊𝖉𝖆 ｡ (@lindseybieda) <a href="https://twitter.com/lindseybieda/status/969296749985779712?ref_src=twsrc%5Etfw">March
1, 2018</a></blockquote>
<script async="" src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I saw this tweet on my timeline yesterday and I think it really describes my
current mood. The code changes over time and I should spend more time making it
flexible enough to support this continues to grow. Abstraction is not the right
way.</p>
<p>So, passionate code engineer always abstract? That’s not the giveaway, Java
engineer always abstract? maybe.</p>
Review book Site Reliability Engineering2018-03-15T10:08:27+00:00http://gianarb.it/blog/site-reliability-engineering-review<p>I bought <a href="http://amzn.to/2pfeHBU">Site Reliability Engineering</a> a lot of months
ago. I read the ebook first but I am the kind of people that buy also paper
books when they are good, so if you are working on a distributed and scalable
environment it’s something that you should read.</p>
<p>Published by O’Reilly and edited by Betsy Beyer, Chris Jones, Jennifer Petoff
and Niall Richard Murphy it is written by many Google engineers and it’s about
the experience they made scaling services like Google Maps, Calendar, YouTube
and all the other products.</p>
<p>I spoke with different people about this book and a lot of them told me that
there is nothing new on that. It’s just cool because Google made it cool.</p>
<p>I have a different option. It’s a nice book because it is a complete source of
information about design and processes in a highly scalable environment.
Probably
some of them topic are well known but it’s hard to find all this information in
a single place.</p>
<p><img alt="Site Reliability Engineering book" src="/img/sre-book.jpeg" class="img-fluid" /></p>
<p>To be fair, it has 524 pages so it’s not a fast read. It took me few
months but I keep it around when I need to explain concepts like how to
dimension and measure loads in a services environment. SLA, SLO and how to use
them properly to manage and measure risks are
well explained, circuit breaking and more, in general, a lot of good procedures
about
resiliency, teamwork, delivery are explained in this book.</p>
<p>There is a nice chapter about how to use the metrics to set up a function and
smart alerting system to keep engineer on-call in a safe and comfortable
environment.</p>
<p>Another one about how Google design resilient applications and how they
dimension services. How much and how deep they know their services impressed me
a lot.</p>
<p><strong>Site Reliability Engineering</strong> is a good mix of concepts that you can apply
through your day to day no-at-Google job and the all the
Google scale “freaky fun”.</p>
<p>So, in the end, I will define like the bible for an engineer that wish to work
in a high-scalable environment. Doesn’t matter if you are not there yet or if
you
won’t serve millions of requests per second. It’s good to read and to keep
around.</p>
<p>The <a href="https://landing.google.com/sre/book/index.html">HTML version</a> of this book
is now available online for free.</p>
What is distributed tracing. Zoom on opencensus and opentracing2018-02-18T10:08:27+00:00http://gianarb.it/blog/what-is-distributed-tracing-opentracing-opencensus<p>A few months ago I started to actively study, support and use opentracing and
more in general the distributed tracing topic.</p>
<p>In this article, I will share something about what I have learned starting from
the basic I hope to get your thoughts and questions via Twitter
<a href="https://twitter.com/gianarb">@gianarb</a>.</p>
<p>We all know the trend for the last couple of years. Spread applications across
containers, cloud providers and split them into smallest units called services
or microservices, pets…</p>
<p>This procedure brings a lot of advantages:</p>
<ul>
<li>you can manage people in a better way and spread them across this small units.</li>
<li>small units are easy to understand for new people or after a couple of months.
In a work like our where there is a high turnover having the ability to
rewrite a service if nobody knows it in a couple of days it’s great.</li>
<li>You can monitor these units in a better way and if you detect scalability
problems or bottleneck you can stay focused on the specific problem without
having other functions around. It enforces the single responsibility in some
way.</li>
</ul>
<p>Btw there are other points for sure, but the last one is very important and I
think it helps us to understand why tracing is so important now.</p>
<p>We discover that monitor this pets is very hard and it’s different compared to
the previous situation. A lot of teams discovered this complexity moving forward
with services making noise in production.</p>
<p>Our focus is not on the virtual machine, on the hostname or the even on the
container. I don’t care about the status of the server. I care about the status
of the service and even deeper I care about the status of a single event in my
system. This is also one of the reasons why tools like Kafka are so powerful and
popular. Reply a section of your history and collect events like user
registration, new invoice, new attendees register at your event, new flight
booked or new article published are the most interesting part here.</p>
<p>Servers, containers should be replaceable things and they shouldn’t be a
problem. The core here is the event. And you need to be 100% sure about having
it in someplace.</p>
<p>Same for monitoring, if the servers, containers are not important but events are
you should monitor the event and not the server.</p>
<p>Oh, don’t forget about distribution. It makes everything worst and more
complicated my dear. Events move faster than everything else. They are across
services, containers, data centers.</p>
<p>Where is the event? Where it failed. How a spike for particular events behave on
your system? If you have too many new registrations are you still able to serve
your applications?</p>
<p>In a big distributed environments what a particular service is calling? Or is it
used? Maybe nobody is using it anymore. These questions need to have an answer.</p>
<p>Distributed tracing is one of the ways. It doesn’t solve all the problems but it
provides a new point of view.</p>
<p>In practice speaking in HTTP terms tracings translate on following a specific
request from its start (mobile app, web app, cronjobs, other apps) so it’s the .</p>
<p>Registering how many applications it crosses to, for how long. Labeling these
metrics you can event understand latency between services.</p>
<p><img src="http://www.hawkular.org/img/blog/2017/2017-04-19-jaeger-trace.png" class="img-fluid" />
<small>from http://www.hawkular.org/</small></p>
<p>Speaking in the right language, this image describes a trace. It’s an HTTP to
<code class="highlighter-rouge">fronted</code> service. It’s a GET request on the <code class="highlighter-rouge">/dispatch</code> route. You can see how
far you can go. A trace is a collection of spans.</p>
<p>Every span has it’s own id and an optional parent id to create the hierarchy.
Spans support what is called Span Tags. It is a key-value store where the key is
always a string and some of them are “reserved” to describe specific behaviors.
You can look at them <a href="https://github.com/opentracing/specification/blob/master/semantic_conventions.md#standard-span-tags-and-log-fields">inside the specification
itselt</a>.
Usually, UI is using this standard tag to build a nice visualization. For
example if a span contains the tag <code class="highlighter-rouge">error</code> a lot of tracers colored it red.</p>
<p>I suggest you read at the standard tags because it will give you the idea about
how descriptive a span can be.</p>
<p>The architecture looks like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> +-------------+ +---------+ +----------+ +------------+
| Application | | Library | | OSS | | RPC/IPC |
| Code | | Code | | Services | | Frameworks |
+-------------+ +---------+ +----------+ +------------+
| | | |
| | | |
v v v v
+-----------------------------------------------------+
| · · · · · · · · · · OpenTracing · · · · · · · · · · |
+-----------------------------------------------------+
| | | |
| | | |
v v v v
+-----------+ +-------------+ +-------------+ +-----------+
| Tracing | | Logging | | Metrics | | Tracing |
| System A | | Framework B | | Framework C | | System D |
+-----------+ +-------------+ +-------------+ +-----------+
</code></pre></div></div>
<p><small>from <a href="http://opentracing.io/documentation/pages/instrumentation/common-use-cases.html" target="_blank">opentracing.org</a><small></small></small></p>
<p>There are different instrumentation libraries across multiple languages and you
need to embed one of them in your application. It usually provides a global
variable where you can add spans too. Time by time they are stored in the
tracer that you select. If you are using Zipkin as tracer you can select
different backends like ElasticSearch and Cassandra.
Tracers provides API and UI to store and visualize traces.</p>
<p>As you can see from the graph above Opentracing “is able” to push to Tracers,
Logging system, metrics and so on. With my experience with opentracing, I don’t
know how this can be done.</p>
<p>I always used it with a Tracer like Zipkin or Jaeger to store spans. Logs are
covered by the spec because you can attach to every spans one or multiple <code class="highlighter-rouge">Span
Logs</code>.</p>
<blockquote>
<p>each of which is itself a key:value map paired with a timestamp. The keys must
be strings, though the values may be of any type. Not all OpenTracing
implementations must support every value type.</p>
</blockquote>
<p><small>from <a href="https://github.com/opentracing/specification/blob/master/specification.md" target="_blank">opentracing.org</a><small></small></small></p>
<p>The idea behind this feature is clear. There are too many buzzwords: metrics,
logs, events, time series and now traces.</p>
<p>It’s easy to end up with more
instrumentation libraries that business code. That’s probably why opentracing
cover this uses case. Logs and traces are time series. That’s probably why
metrics are there.</p>
<p>Using the go-sdk it looks like this:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">span</span><span class="p">,</span><span class="x"> </span><span class="n">ctx</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">opentracing</span><span class="o">.</span><span class="n">StartSpanFromContext</span><span class="p">(</span><span class="n">ctx</span><span class="p">,</span><span class="x"> </span><span class="s">"operation_name"</span><span class="p">)</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="n">span</span><span class="o">.</span><span class="n">Finish</span><span class="p">()</span><span class="x">
</span><span class="n">span</span><span class="o">.</span><span class="n">LogFields</span><span class="p">(</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"event"</span><span class="p">,</span><span class="x"> </span><span class="s">"soft error"</span><span class="p">),</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">String</span><span class="p">(</span><span class="s">"type"</span><span class="p">,</span><span class="x"> </span><span class="s">"cache timeout"</span><span class="p">),</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Int</span><span class="p">(</span><span class="s">"waited.millis"</span><span class="p">,</span><span class="x"> </span><span class="m">1500</span><span class="p">))</span><span class="x">
</span></code></pre></div></div>
<p>But I am not able to find a way to say: “Forward all these logs to ….elastic
and this traces to Zipkin”. And I don’t know if the expectation is to have
tracers smart enough to do that. But from my experience trying to extend Zipkin,
this looks like a hard idea. At first, because the tracers are out of the
OpenTracing’s scope.</p>
<p>If the goal is to wrap together everything logs have precise use case from ages.
They work pretty well and you can’t change the expectation. They can be a
real-time stream on stdout, stderr and/or other thousands of exporter. I can’t
find this kind of work there. So, looking at the code it’s not clear who is in
charge of what. But the graph is pretty.</p>
<p>I like the idea and I started looking at <a href="https://opencensus.io/">OpenCensus</a> a
library open sourced by Google from its experience with StackDriver and the
Google’s scale. It has its
<a href="https://github.com/census-instrumentation/opencensus-specs">specification</a> and
it provides a set of <a href="https://github.com/census-instrumentation/">libraries</a>
that you can add to your application to get what they call stats, traces out
from your app. Stat stays for metrics, events. It’s another buzz probably!</p>
<p>The concept looks similar to OpenTracing, obviously, the specs are different.</p>
<p>Looking at the code, the go-SDK looks a lot more clear. I can clearly see stats
and tracing objects, they both accept exporters and they can be Prometheus,
Zipkin, Jaeger, StackDriver and so on. I like the idea that the exporter is part
of the project, you don’t need a tracing application like Zipkin, you can write
your exporter to store data in your custom database and you are ready to go.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.
├── appveyor.yml
├── exporter
│ ├── jaeger
│ ├── prometheus
│ ├── stackdriver
│ └── zipkin
├── internal
├── plugin
├── README.md
├── stats
│ ├── internal
│ ├── ...
│ └── view
├── tag
├── trace
</code></pre></div></div>
<p>You can probably do the same with OpenTracing writing your tracer that store
things in your custom databases jumping Zipkin and Jaeger, it looks a bit more
complicated looking at the interface:</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">// opencensus-go/trace/export.go</span><span class="x">
</span><span class="c">// Exporter is a type for functions that receive sampled trace spans.</span><span class="x">
</span><span class="c">//</span><span class="x">
</span><span class="c">// The ExportSpan method should be safe for concurrent use and should return</span><span class="x">
</span><span class="c">// quickly; if an Exporter takes a significant amount of time to process a</span><span class="x">
</span><span class="c">// SpanData, that work should be done on another goroutine.</span><span class="x">
</span><span class="c">//</span><span class="x">
</span><span class="c">// The SpanData should not be modified, but a pointer to it can be kept.</span><span class="x">
</span><span class="k">type</span><span class="x"> </span><span class="n">Exporter</span><span class="x"> </span><span class="k">interface</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">ExportSpan</span><span class="p">(</span><span class="n">s</span><span class="x"> </span><span class="o">*</span><span class="n">SpanData</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>// opentracing tracer
type Tracer interface {
// Create, start, and return a new Span with the given `operationName` and
// incorporate the given StartSpanOption `opts`. (Note that `opts` borrows
// from the "functional options" pattern, per
// http://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis)
//
// A Span with no SpanReference options (e.g., opentracing.ChildOf() or
// opentracing.FollowsFrom()) becomes the root of its own trace.
//
StartSpan(operationName string, opts ...StartSpanOption) Span
// Inject() takes the `sm` SpanContext instance and injects it for
// propagation within `carrier`. The actual type of `carrier` depends on
// the value of `format`.
//
// OpenTracing defines a common set of `format` values (see BuiltinFormat),
// and each has an expected carrier type.
//
// Other packages may declare their own `format` values, much like the keys
// used by `context.Context` (see
// https://godoc.org/golang.org/x/net/context#WithValue).
//
Inject(sm SpanContext, format interface{}, carrier interface{}) error
// Extract() returns a SpanContext instance given `format` and `carrier`.
//
// OpenTracing defines a common set of `format` values (see BuiltinFormat),
// and each has an expected carrier type.
//
// Other packages may declare their own `format` values, much like the keys
// used by `context.Context` (see
// https://godoc.org/golang.org/x/net/context#WithValue).
//
Extract(format interface{}, carrier interface{}) (SpanContext, error)
}
</code></pre></div></div>
<p>OpenTracing doesn’t care about exporter and tracers, something else handle that
complexity, (the user, me.. bored) the standard only offers interfaces. I don’t
know if this is good. It really looks a lot more like a common interface
between traces. I like the idea, but I need a lot more.</p>
<p>Now, writing this article I understood that I have a lot more to figure out
about this projects, sadly I realized that in practice they are even more
similar compared my feeling before writing all this down.</p>
<p>Tracing, metrics and instrumentation libraries remain crucial from my point of
view. You can write everything you want but if you are not able to understand
what’s happening you are not making a good job. You look like a monkey.</p>
<p>Personal I would like to find a common and good library to wrap together all
the buzzwords stats, spans, traces, metrics, time series, logs because they are
all the same concept just from a different point of view.</p>
<p>Everything is a point in time, grouped, ordered or with a specific hierarchy.
You can use them as aggregate, to compare, to alert and so on. A powerful
implementation should be able to combine both needs an easy ingestion with a
clear output.</p>
<p>I think that OpenTracing has a lot to do from both sides in and out. OpenCensus
looks good from an ingestion point of view. Nothing about logs in OpenCensus
maybe because they are good enough as they are but we need to be able to cross
reference logs, traces, metrics from infrastructure and application events from
dashboards and automatic tools.</p>
<p>It looks like, with both setup, that you still need a platform capable to serve
and use this data. A lot of people will answer that it’s out of scope for
these projects, but I am pretty sure we all learned that just storing events is
not enough.</p>
The right balance2018-02-09T10:08:27+00:00http://gianarb.it/blog/the-development-balance<p>My daily job as a developer other than programming is about finding the right
balance between different things:</p>
<ol>
<li>Buzzword driven development vs rock solid boring things.</li>
<li>New technologies vs something already adopted inside the company.</li>
<li>State of the art implementation. Something that can work in an hour.</li>
</ol>
<p>And many more. I am sure you will have your own list. What I noticed during
these years working for different companies, with different teams and in
different states is that there are a lot of kind of developers, we are full of
companies and projects to develop.</p>
<p>You should really look for the right place for you, but you need to know what to
look for.</p>
<p>What I am trying to say that people, colleagues, companies can help you to find
your balance be proactive on this research. Speak with your manager or with your
colleagues about what you are happy to do or not you will find interesting
answers if you are working with people with your same mindset. You will probably
come up selling and buying tasks and issues from member of the teams because
they like more what you are doing and vice versa.</p>
<p>Sometime you will even have the sensibility to grab a bored task just to have it
done and leave your colleagues free to do something more fun.</p>
<p>This is the kind of work that I like. I am almost sure about that now. A place
where shit needs to be done and you have an active word on how, it doesn’t need
to be a decision, sometime I don’t know what’s better for the company or for the
project, but can’t be always a black box that come from the ground and needs to
be done. I am not an check list implementer and dealing with my colleagues and
other people is an active and very good part of my day.</p>
<p>When it’s time to have fun because things are going well and even when you are
the unique different voice out form a meeting. Find the right place where you
feel free to say your opinion, be wise and mature enough to know that your
opinion can be good or bad can’t be always the one to follow.</p>
<p>The right balance between all of these things, across teams, companies gives to
me the feel to be in the right place.</p>
Kubernetes up and running2017-12-19T10:08:27+00:00http://gianarb.it/blog/kubernetes-up-and-running<p>I read <a href="http://amzn.to/2zflChj">“Kubernetes up and running”</a> an O’Reilly book
written by Kelsey Hightower, Brendan Burns and Joe Beda.</p>
<p>It looks like the instrumental manual, you look for it when you buy something.
Based on your knowledge of it you read or not that manual.</p>
<p>I have a good knowledge of containers, orchestrator and cloud computing but I
never worked with Kubernetes until 2 weeks ago when I started a co-managed k8s
cluster on scaleway with <a href="https://twitter.com/fntlnz">Lorenzo</a>.</p>
<p>The book is well written, I read it in less than one week. The chapters are well
split because I was able to jump “Builiding a Raspberry Pi Kubernetes cluster”
because I am not really interested without any pain.</p>
<p>Chapters like “Deploying real world applications”, “Service Discovery” are good
and the book covers all the basic concepts that you need to know about
Kubernetes. You can feel all the experience that the three authors have on the
topic. There are gold feedbacks about what they learned using and building what
is now the orchestration standard.</p>
<p>Just to summarize, if you are using kubernetes and you like papers, this book is
a good way to have a documentation on paper. If you are new to Kubernetes is the
best way to start.</p>
<p>Thanks to all the authors! If you have any questions let me know
<a href="https://twitter.com/gianarb">@gianarb</a></p>
Desk setup2017-12-17T10:08:27+00:00http://gianarb.it/blog/desk-setup<p>As you probably know in April 2017 I moved back after an year and half in Dublin
and I started to work from home as SRE at InfluxData.</p>
<p>I am ready to write a small post about my current setup.</p>
<h2 id="ikea-markus">Ikea Markus</h2>
<p>First things I bought an <a href="http://www.ikea.com/gb/en/products/chairs-stools-benches/office-chairs/markus-swivel-chair-glose-black-art-20103101/">Ikea
Markus</a>.
It is comfortable and it has a competitive price. It’s flexible and well designed.</p>
<p>I don’t have a lot to say about it. If you are not passionate about expensive
and weird chairs you can go for this one. It will work!</p>
<h2 id="stand-mount">Stand Mount</h2>
<p>My setup counts two boring Asus monitor, one horizontal and one vertical,
Lorenzo suggested to me this <a href="http://amzn.to/2yMe59C">standmount support for two monitor</a>.</p>
<p>Day by day I discover how bad I am using more than one monitor. Change focus so
often is not for me but I like the vertical monitor when I am debugging some
weird application.</p>
<h2 id="asus-zenbook-3">Asus Zenbook 3</h2>
<p>I have an <a href="http://amzn.to/2AHAy9N">Asus Zenbook 3</a> the only usb-c is kind of a
pain. I am a traveler and a speaker. I don’t enjoy its low weight (900gr) that
often because I always need some adapter.</p>
<p>For traveling the adapter <a href="http://amzn.to/2CKIMPG">Asus Universal Dock</a> is good. It
is embedded with a charger it means that you need to have power supply to use
it. I wrote an article about it and I was very disappointed about the product.
But now that I am using it only for traveling purpose it’s not too bad.</p>
<p>If you are a multi desktop user you need to remember that it doesn’t have an
external video card, it has VGA and HDMI but you can use only one of them at the
time.</p>
<p>I used Ubuntu 17.04 and 17.10. Now I am using ArchLinux and both laptop and
Universal Dock need to install some drivers, to work a bit on audio
configuration and so on. But it’s a good challenge and almost everything works
out of the box.</p>
<h2 id="logitech-c922">Logitech C922</h2>
<p>The embedded Asus WebCam is not great. If you are looking for high definition or
an acceptable quality you need to have an external webcam.</p>
<p>I work from and when I have a meeting with colleagues and friends I would like
to offer to them good experience.</p>
<p>The <a href="http://amzn.to/2kEnJ9o">Logitech C922</a> is not powerful enough to make me
beautiful but it makes an amazing work and it’s very good.</p>
<p>If you record tutorials or workshop you should think about having one of this.
It comes with a small tripod to setup it where ever you like.</p>
<h2 id="usb-c-adapter">USB-C adapter</h2>
<p>As I told you the world is not ready for the USB-C but I am!
<a href="http://amzn.to/2zhPbSQ">Plugable</a> makes my life very simple.</p>
<p>WebCam, two monitors, Ethernet cable are always attached to it and I just need
to plug my laptop in via the USB-C and everything will happen.</p>
<p>It’s an expensive toy but I am using it on Linux and it’s working. The company
doesn’t officially support it but there is a
<a href="https://github.com/displaylink/evdi">DisplayLink</a> driver open source on GitHub
that you can use.</p>
<h2 id="desk">Desk</h2>
<p>Last but not least I have a standing desk.</p>
<p>I think a good chair, gym, swim are better solution to keep you healthy but I
change my point of view and my position help me to stay focused.</p>
<p>Every time I have a boring or complex task at some point just toggle my
current position from down to up or vice versa gives me some fresh power to
end it well.</p>
<p>I monitored the <a href="https://www.ikea.com/gb/en/products/desks/office-desks/bekant-desk-sit-stand-oak-veneer-black-spr-29061187/">Ikea
Bekant</a>
for a lot of months but I was not sure about investing money a standind desk.</p>
<p>I looked at them for so long that Ikea started a very good discount campaign and
I just bought it. I took only the mechanical legs because I like the feeling of
real wood and I bought the table separately.</p>
<p>That’s it! Bye!</p>
From Docker to Moby2017-10-20T10:08:27+00:00http://gianarb.it/blog/from-docker-to-moby<p>At DockerCon 2017 Austin
<a href="https://blog.docker.com/2017/04/introducing-the-moby-project/">Moby</a> was the
big announcement.</p>
<p>It created confusion and some communities are still trying to understand what is
going on. I think it’s time to step back and see what we have after seven months
after the announcement.</p>
<ol>
<li><code class="highlighter-rouge">containerd</code> is living a new life, the first stable release will happen soon.
It has been donated to CNCF.</li>
<li><code class="highlighter-rouge">notary</code> is the project behind <code class="highlighter-rouge">docker trust</code>. I wrote a full e-book about
<a href="https://scaledocker.com">Docker Security</a> if you need to know more. This
also has been donated to the CNCF.</li>
<li>github.com/docker/docker doesn’t exist anymore there is a new repository
called github.com/moby/moby .</li>
<li><a href="https://github.com/docker/cli">CLI</a> has a separate home.</li>
<li>docker-ce is the first example of moby assembling. It is made my Docker Inc.</li>
</ol>
<p>Containers are not a first class citizen in Linux.</p>
<p><img class="img-fluid" src="/img/container-is-not-real.jpeg" /></p>
<p>They are a combination of cgroups, namespaces and other kernel features. They are
also there from a lot of year. LXD is one of the first project that mentioned
container but the API wasn’t really friendly and only few people are using it.</p>
<p>Docker created a clean and usable api that human beings are happy to use. It
created an ecosystem with an amazing and complete UX. Distribution, Dockerfile,
<code class="highlighter-rouge">docker run</code>, <code class="highlighter-rouge">docker image</code> and so on.</p>
<p>That’s what Docker is, in my opinion. Other than a great community and a fast
growing company.</p>
<p>What Docker is doing with Moby is to give the ability to competitors, startups, new
projects to join the ecosystem that we built in all these 4 years.</p>
<p>Moby in other hands is giving the ability at Docker to take ownership of the
clean and usable experience. The <code class="highlighter-rouge">Docker CLI</code> that we know and use every day
will stay open source, but not the moby project’s part. It will be owned by
Docker. As I wrote above, the code is already moved out.</p>
<p>Moby allows other companies and organisations to build their
user interface based on what they need. Or to build their product on top of a
open source project designed to be modular.</p>
<p>Cloud and container moves fast Amazon with ECS, RedHat with OpenShift,
Pivotal with Cloud Foundry, Mesos with Mesosphere, Microsoft with Azure
Container Service, Docker with Docker, they are all pushing hard to build
projects around containers to sell them at big and small corporations to make
legacy projects less bored.</p>
<blockquote>
<p>Legacy is the new buzzword</p>
</blockquote>
<p>Docker will continue to assemble and ship docker as we know it. The project is
called <code class="highlighter-rouge">docker-ce</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get install docker-ce
docker run -p 80:80 nginx:latest
</code></pre></div></div>
<p>Everything happens down the street, in the open source ecosystem. Moby won’t
contain the CLI that we know.</p>
<p>Moby won’t have the swarmkit integration as we know it. It was something that
Docker as company was looking to have. Mainly to inject an orchestrator in
million of laptops. Other companies and projects that are not using swarm don’t
need it and they will be able to remove it in some way.</p>
<p>Companies like Pivotal, AWS are working on
<code class="highlighter-rouge">containerd</code> because other the runtime behind Docker it’s what matters for a lot
of projects that are just looking to run containers without all the layers on
top of it to make it friendly. ECS, Cloud Foundry are the actual layers on top
of “what runs a container”.</p>
<p>Container orchestrator doesn’t really care about how or who spins up a container,
they just need to know that there is something able to do that.</p>
<p>It is what Kubernetes does with CRI. They don’t care about Docker, CRI-O,
containerd. It’s out of scope they just need a common interface. In this case is
a gRPC interface that every runtime should implement. Here a list of them:</p>
<ul>
<li><a href="https://github.com/kubernetes-incubator/cri-o">cri-o</a></li>
<li><a href="https://github.com/kubernetes-incubator/cri-containerd">cri-containerd</a></li>
<li><a href="https://github.com/kubernetes-incubator/rktlet">rktlet</a></li>
</ul>
<p>That’s a subset of reasons about why everything is happening:</p>
<ul>
<li>Docker Inc. will be free to iterate on their business services and projects
without breaking every application in the world. And they will have more
flexibility on what they can do as company.</li>
<li>The transaction between Docker to Moby is the perfect chance to split the
project to different repositories we already spoke about docker-cli, containerd
and so on.</li>
<li>Separation of concern is popular design pattern. Split
projects on smallest libraries allow us to be focused on one specific scope of the
project at the time.
<a href="https://github.com/moby/buildkit">buildkit</a> is the perfect example. It’s the
evolution of the <code class="highlighter-rouge">docker build</code> command. We had a demo at the MobySummit and
it looks amazing!</li>
</ul>
<p>That’s almost it. Let’s summarise:</p>
<p><strong>Are you a company in the container movement?</strong>
You are competing with Docker building container things and you was complaining
about them breaking compatibility or things like that now you should blame the
Moby community.</p>
<p><strong>Are you using docker run?</strong>
You are fine! You will be able to do what you was doing before.</p>
<p><strong>Are you a OpenSource guru?</strong>
Maybe you will be a bit disappointed if you worked hard on docker-cli and now
Docker will bring your code out but you signed a CLA, the CLI will stay open
source. Blame yourself.</p>
<p>That’s it! Or at least that’s what I understood.</p>
Git git git, but better2017-10-10T10:08:27+00:00http://gianarb.it/blog/git-git-git-but-better<p>I can’t say that Git is a new topic. Find somebody unable to explain how a
version control system was working was very hard. Now it’s almost impossible.</p>
<p>I used SVN and Git for many years and I also put together some unusual use case
for example: <a href="https://devzone.zend.com/6134/splitting-zend-framework-using-the-cloud/">“Splitting Zend Framework Using the Cloud”</a>
is a project that I made with Corley SRL my previous company and the Zend
Framework team.</p>
<p>It helped me to put my hands down on the Git file system and I discovered a lot
of features and capabilities that are not the usual: commit, checkout, reset,
branch, cherry-pick, rebase and so on.</p>
<p>But during my experience building cloud at InfluxData I need to say that I can
see a change of my mindset, I am sharing this because I am kind of proud of
this. It’s probably not super good looking at the time required to achieve this
goals but how cares!</p>
<blockquote>
<p>Sometimes it’s the journey that teaches you a lot about your destination.
(Drake)</p>
</blockquote>
<p>I don’t know this Drake, I am not even sure if it’s the right author of the
quote but that’s not the point.</p>
<p>At InfluxData, just to give you more context, I am working on a sort of
scheduler that provisions and orchestrate servers and containers on our <a href="https://cloud.influxdata.com/">cloud
product</a>. A lot of CoreOS instances, go, Docker and
AWS api calls.</p>
<p>It’s a modest codebase in terms of size but it is keeping up a huge amount of
servers, I am actively working on the code base almost by myself and I am kind
of enjoying this. Nate, Goller and all the teams are supporting my approach and
are using it but I am not using Git because hundreds of developers need to
collaborate on the same line of code. I had some experience in that environment
working as contributor in many open source project. But this time is different.</p>
<p>I am mainly alone on a codebase that I didn’t start and I don’t know very well,
this project is running in production on a good amount of EC2.</p>
<p>I really love the idea of having a clean and readable Git history. I am not
saying that because it’s cool. I am saying that because every time I commit my
code I am thinking about which file to add/commit <code class="highlighter-rouge">-a</code> is not really an option
that I use that much anymore. I think about the title and the message.</p>
<p>I try to avoid the <code class="highlighter-rouge">WIP</code> message and I use it only if I am sure about a future
squash, rebase and if I need to push my code to ask for ideas and options (as I
said I am writing code almost alone, but I am always looking for support from my
great co-workers).</p>
<p>This has a very big value I think also as remote worker. This is my first
experience in this environment and for a no-native English a good and
self explanatory title can be the hardest part of the work but it will help
other people to understand what I am doing.</p>
<p>When you are working on a new codebase and you have tasks that require
refactoring to be achieved in a fancy and professional way you will find
yourself moving code around without really be able to figure out when and how it
will become useful to close your task and open the PR that your team lead is
waiting for. At the end if you start to write code and you commit your changes
at the end of the day as I was doing at the beginning after a couple of days you
will figure out that your PR is too big and you are scared to merge them.
And probably it’s just the PR that is preparing the codebase to get the initial
requests. I hated the situation but if you think about what I wrote you will
find that it’s totally wrong.</p>
<p>VCS is not there as saving point, you are not plaining Crash Bandicoot anymore,
you don’t need to use Git as your personal “ooga booga”. The right commit
contains an atomic information about a feature, bug fix or whatever.</p>
<p><img src="/img/crash_bandcioot.jpg" alt="" /></p>
<p>These are the questions that I am asking myself now before to make a commit:</p>
<ul>
<li>am I confident cherry-picking this commit to <code class="highlighter-rouge">master</code>? This is a good way to
make your commit small and easy to merge. If one of your PR is becoming too
big and you have “cherry-picked” commits you can select some of them merge
them as single PR.</li>
<li>are deploy and rollback easy actions? This is similar to previous one but I am
the one that deploy and monitor the service in production. I need to ask this
question to myself before every merged PR.</li>
<li>Looking at the name of the branch that in my case the task in my
viewfinder the commit that I am creating is about it or can I create a new PR
just for this piece of code? This helps me a log to split my PR and to them
small. A small PR is easier to review, it has a better scope and it makes me
less scared to deploy it.</li>
</ul>
<p>Git is more than a couple of commands that you can execute. You need to
be in the right mindset to enjoy all the power.</p>
Orbiter the Docker Swarm autoscaler on the road to BETA-12017-08-09T10:08:27+00:00http://gianarb.it/blog/orbiter-the-swarm-autoscaler-moves<p>Orbiter is an open source project written in go hosted on
<a href="https://github.com/gianarb/orbiter">GitHub</a>. It provides autoscaling
capabilities in your Docker Swarm Cluster.</p>
<p>As you probably know at the moment autoscaling is not a feature supported
natively by Docker Swarm but this is not a problem at all.</p>
<p>Docker Swarm provides a useful API that helps you improving its capabilities.</p>
<p>I created Orbiter months ago as use case with InfluxDB and to allow services to
scale automatically based on signal <code class="highlighter-rouge">up</code> or <code class="highlighter-rouge">down</code>. You can follow the webinar
that I made with InfluDB
<a href="https://www.influxdata.com/resources/influxdata-helps-docker-auto-scale-monitoring/?ao_campid=70137000000Jgw7">here</a>.</p>
<p>This article is not about “How it works”. You can <a href="https://gianarb.it/blog/orbiter-docker-swarm-autoscaler">read more here about how it
works</a> and you can
watch the embedded video that I made in the Docker HQ in San Francisco.</p>
<p>Yesterday we made some very good improvements and we are moving forward to tag
the first beta release. I need to say a big thanks to <a href="https://github.com/mbovo">Manuel
Bovo</a>. He coded pretty much all the features listed
here.</p>
<ol>
<li>
<p><a href="https://github.com/gianarb/orbiter/pull/26">PR #26</a> e2e working example. <a href="https://github.com/gianarb/orbiter/tree/master/contrib/swarm">Please try
it</a>.</p>
</li>
<li>
<p><a href="https://github.com/gianarb/orbiter/pull/27">PR #27</a> Now Orbiter has
background job that listen on the Docker Swarm event API and register and
de-register new services <a href="https://github.com/gianarb/orbiter#autodetect">deployed with right
labels</a>. You don’t need to
restart orbiter anymore. It detect new services automatically.</p>
</li>
<li>
<p><a href="https://github.com/gianarb/orbiter/pull/29">PR #29</a> Fixed the up/down range.
Now we can not scale under 1 tasks but we can scale up services with 0 tasks.</p>
</li>
<li>
<p><a href="https://github.com/gianarb/orbiter/pull/31">PR #31</a> We have a cooldown
period configurable via label <code class="highlighter-rouge">orbiter.cooldown</code>. This fix avoid multiple
scaling in a short amount of time.</p>
</li>
<li>
<p><a href="https://github.com/gianarb/orbiter/pull/32">PR #32</a> We are migrating our API
base root. Now all the API are <code class="highlighter-rouge">/v1/orbiter/.....</code>. At the moment we are
supporting old and new routes. <strong>In October I will remove the old one. Please
migrate to <code class="highlighter-rouge">/v1/orbiter/....</code> now!</strong>.</p>
</li>
</ol>
<h2 id="now">Now?</h2>
<p>That’s a good question, but I have part of the answer. In October the plan is to
release a BETA and finally the first stable version but what we need to do to go
there?</p>
<ul>
<li>Offer a proper auth method. Manuel started this
<a href="https://github.com/gianarb/orbiter/pull/33">PR</a>. I have some concerns but
we are on the right path.</li>
<li>Make orbiter “Only-Swarm”. The project started with the vision to become a
general purpose autoscaler. But this is not in line with the idea of single
responsibility and we designed a very clean API for Docker Swarm, make it
usable in other context is not going to work. We tried it with DigitalOcean
but the api and the project looks too complex and I love simplicity.</li>
<li>Get other feedback from the community to merge valuable features before the
stable release.</li>
</ul>
<p>That’s it! Share it and give it a try! For any question I am available on
twitter (@gianarb) or open an issue.</p>
Asus universal dock station driver2017-08-03T10:08:27+00:00http://gianarb.it/blog/asus-universal-dock-driver<p>Every developer loves to share things about it’s setup. They also loves to make
it better and to spend time on it.</p>
<p>Lorenzo <a href="https://twitter.com/fntlnz">(fntlnz)</a> is super on it! I am
not, plus I bought a Zenbook 3. Super slim, less than 1kg, I can use it to cut
ham probably but the unique USB-C is driving me crazy.</p>
<p>Probably more than the actual 40 degrees that I have in my home office now!
It is probably why I am writing this post btw.</p>
<p>When I bought this laptop 7 months ago the Universal Docker Station was not
available and I wasn’t even able to install linux on this laptop.</p>
<p>Now I have an <a href="https://www.asus.com/Laptops-Accessory/Universal-Dock/">Asus Universal Dock
station</a>. I am feeling a
little bit better but to work it replace a normal charger, it means that without
a socket near you, I can not use a USB… Amazing experience.</p>
<p>I tried other adapter but I didn’t find one good enough. Every one of them had
some input or output port unusable for some reason. Most of them because the
BIOS has a different watt limit and they can not charge the laptop. I never
received a response from ASUS about it. That’s great.</p>
<p>Anyway I am writing this article just as note for myself about the driver that
Lorenzo discover to have the Asus Universal Dock Station’s ethernet port
running.</p>
<p><a href="http://www.realtek.com/DOWNLOADS/downloadsView.aspx?Langid=1&amp;PNid=13&amp;PFid=5&amp;Level=5&amp;Conn=4&amp;DownTypeID=3&amp;GetDown=false">Realtek ethernet
driver</a>.
It’s super easy to install. Just compile it and it will work.</p>
CNCF Italy, first event about opentracing2017-06-05T10:08:27+00:00http://gianarb.it/blog/cncf-italy-first-event<p>CNCF is a branch of The Linux Foundation focused on Cloud Computing and modern
scalable architectures. it’s supporting tools like Kubernetes, Prometheus,
containerd and so on. If you are using one of them or you are looking to know
more about them, this is your meetup. Join us! hashtag #CNCFItaly on twitter.</p>
<p>The event will be 13th July 19.00pm at Toolboox Office in Turin. Reserve your
seat on <a href="https://www.meetup.com/CNCF-Italy/events/241118593/">Meetup.com</a>.</p>
<iframe src="https://www.google.com/maps/embed?pb=!1m18!1m12!1m3!1d2818.7526155267037!2d7.667091951510242!3d45.05024176888683!2m3!1f0!2f0!3f0!3m2!1i1024!2i768!4f13.1!3m3!1m2!1s0x47886d37dd5ababd%3A0x2adc0b0e358ddb6c!2sToolbox+Coworking!5e0!3m2!1sit!2sit!4v1499676857774" width="600" height="450" frameborder="0" style="border:0" allowfullscreen=""></iframe>
<p>This will be a full evening about OpenTracing. OpenTracing <a href="https://www.cncf.io/blog/2016/10/20/opentracing-turning-the-lights-on-for-microservices/">turns the light on
for
microservices</a>.
It is a specification to store and manage trace. How can we follow what’s going
on from the beginning to the end of our requests? What’s happening when they
cross different services? Where is the bottleneck? Tracing helps you to
understand what’s going on. It’s not just for microservices but also for
caching, queue system an so on. Have a <a href="https://trends.google.it/trends/explore?q=opentracing">look at the
trends</a> we need to know
more about it!</p>
<p>Beers, Pizza are offered by CNCF after the two sessions!</p>
<p>Other links:</p>
<ul>
<li><a href="https://www.cncf.io/">CNCF.io</a></li>
<li><a href="http://opentracing.io/">Opentracing</a></li>
<li><a href="https://github.com/openzipkin">OpenZipkin by twitter</a></li>
<li><a href="https://www.youtube.com/watch?v=n8mUiLIXkto">Keynote: OpenTracing and Containers: Depth, Breadth, and the Future of
Tracing - Ben Sigelman</a></li>
</ul>
<h2 id="all-done">All done!</h2>
<p>Amazing event! here some pictures and the video is coming soon!</p>
<div class="slide w3-display-container">
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-sponsor-1.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-1.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-5.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-8.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-9.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-10.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-12.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-13.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-14.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-15.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-16.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-17.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-20.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-21.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-22.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-23.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-24.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-25.jpg" />
<img class="mySlides img-fluid" src="/img/cncf-first/Conf-27.jpg" />
<button class="w3-button w3-display-left" onclick="plusDivs(-1)">&#10094;</button>
<button class="w3-button w3-display-right" onclick="plusDivs(+1)">&#10095;</button>
</div>
<style>
.w3-display-left{position:absolute;top:50%;left:0%;transform:translate(0%,-50%);-ms-transform:translate(-0%,-50%)}
.w3-display-right{position:absolute;top:50%;right:0%;transform:translate(0%,-50%);-ms-transform:translate(0%,-50%)}
.w3-tooltip,.w3-display-container{position:relative}.w3-tooltip .w3-text{display:none}.w3-tooltip:hover .w3-text{display:inline-block}
.w3-btn,.w3-button{border:none;display:inline-block;outline:0;padding:8px 16px;vertical-align:middle;overflow:hidden;text-decoration:none;color:inherit;background-color:inherit;text-align:center;cursor:pointer;white-space:nowrap}
.w3-btn:hover{box-shadow:0 8px 16px 0 rgba(0,0,0,0.2),0 6px 20px 0 rgba(0,0,0,0.19)}
.w3-btn,.w3-button{-webkit-touch-callout:none;-webkit-user-select:none;-khtml-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none}
.w3-disabled,.w3-btn:disabled,.w3-button:disabled{cursor:not-allowed;opacity:0.3}.w3-disabled *,:disabled *{pointer-events:none}
.w3-btn.w3-disabled:hover,.w3-btn:disabled:hover{box-shadow:none}
.w3-badge,.w3-tag{background-color:#000;color:#fff;display:inline-block;padding-left:8px;padding-right:8px;text-align:center}.w3-badge{border-radius:50%}
.w3-ul{list-style-type:none;padding:0;margin:0}.w3-ul li{padding:8px 16px;border-bottom:1px solid #ddd}.w3-ul li:last-child{border-bottom:none}
.w3-tooltip,.w3-display-container{position:relative}.w3-tooltip .w3-text{display:none}.w3-tooltip:hover .w3-text{display:inline-block}
.w3-ripple:active{opacity:0.5}.w3-ripple{transition:opacity 0s}
.w3-input{padding:8px;display:block;border:none;border-bottom:1px solid #ccc;width:100%}
.w3-select{padding:9px 0;width:100%;border:none;border-bottom:1px solid #ccc}
.w3-dropdown-click,.w3-dropdown-hover{position:relative;display:inline-block;cursor:pointer}
.w3-dropdown-hover:hover .w3-dropdown-content{display:block;z-index:1}
.w3-dropdown-hover:first-child,.w3-dropdown-click:hover{background-color:#ccc;color:#000}
.w3-dropdown-hover:hover > .w3-button:first-child,.w3-dropdown-click:hover > .w3-button:first-child{background-color:#ccc;color:#000}
.w3-dropdown-content{cursor:auto;color:#000;background-color:#fff;display:none;position:absolute;min-width:160px;margin:0;padding:0}
.w3-check,.w3-radio{width:24px;height:24px;position:relative;top:6px}
.w3-sidebar{height:100%;width:200px;background-color:#fff;position:fixed!important;z-index:1;overflow:auto}
.w3-bar-block .w3-dropdown-hover,.w3-bar-block .w3-dropdown-click{width:100%}
.w3-bar-block .w3-dropdown-hover .w3-dropdown-content,.w3-bar-block .w3-dropdown-click .w3-dropdown-content{min-width:100%}
.w3-bar-block .w3-dropdown-hover .w3-button,.w3-bar-block .w3-dropdown-click .w3-button{width:100%;text-align:left;padding:8px 16px}
</style>
<script>
var slideIndex = 1;
showDivs(slideIndex);
function plusDivs(n) {
showDivs(slideIndex += n);
}
function showDivs(n) {
var i;
var x = document.getElementsByClassName("mySlides");
if (n > x.length) {slideIndex = 1}
if (n < 1) {slideIndex = x.length} ;
for (i = 0; i < x.length; i++) {
x[i].style.display = "none";
}
x[slideIndex-1].style.display = "block";
}
</script>
Container security and immutability2017-06-05T10:08:27+00:00http://gianarb.it/blog/container-security-immutability<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">55 pages
about how to improve container security. <a href="https://twitter.com/ciliumproject">@ciliumproject</a> <a href="https://twitter.com/hashtag/BPF?src=hash">#BPF</a>, best practices, <a href="https://twitter.com/coreos">@coreos</a> clair, <a href="https://twitter.com/hashtag/apparmor?src=hash">#apparmor</a> <a href="https://t.co/ABiuldYA9b">https://t.co/ABiuldYA9b</a> <a href="https://t.co/61jzWxzb1Y">pic.twitter.com/61jzWxzb1Y</a></p>&mdash; :w
!sudo tee % (@GianArb) <a href="https://twitter.com/GianArb/status/871808740080615424">June 5,
2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Security is a fascinating topic.
It’s part of every aspect of a system.
From your email server to the HTTP body validation of your API system.</p>
<p>It’s also a very human centric topic. You can use the most stronger security
approaches but if your rules are too hard to follow or too complicated to
implement the end users or your colleagues will become the perfect breach to be
exploited by bad people.</p>
<p>In distributed systems there are interesting challenges like:</p>
<ul>
<li>How can we trust the instances part of the system itself? I mean, how can we
trust a new application after a pool scale?</li>
<li>All the traffic generated by the system needs to be locked. The network
topology grows with the number of services that we add but that’s not a good
excuse to leak on responsibility about how we do manage our network.</li>
</ul>
<p>When you design a system you need to think about security from different points
of view:</p>
<ul>
<li>Security needs to be efficient. This seems obvious but it’s always something
to keep in mind.</li>
<li>It needs to be easy to use in development mode. As we said before. If security
is making things slower somebody will turn it off.</li>
<li>If you are good enough to make it easy, it will be easier to enforce a secure behavior.</li>
</ul>
<p>All this concepts are well applied in all different projects built by the Docker
Community.
Notary, swarmkit are just a few examples but if you think about the update
framework (TUF) and the whole set of things happening behind every <code class="highlighter-rouge">docker push</code> and <code class="highlighter-rouge">pull</code>
command you suddenly see a great example on how to make complicated things really easy to use.</p>
<p>I published an ebook that you can download for free <a href="http://scaledocker.com">here</a>.
It contains ~55 pages about Container and Docker Security. In this article I
will share one of the concept expressed in that book, <strong>Immutability</strong>.</p>
<p>Docker containers are in fact immutable. This means that a running container
never changes because in case you need to update it, the best practice is to
create a new container with the updated version of your application and delete
the old one.</p>
<p>This aspect is important from different point of views.</p>
<p>Immutability applied to deploy is a big challenge because it opens the door to a very
different set of release strategies like blue green deployment or canary releases.
Immutability also lowers rollback times because you can probably keep the
old version running for a little longer and switch traffic in case of problems.</p>
<p>It’s also a plus from a scalability and stability point of view. For each deploy
you are in fact using provisioning scripts and build tools to package and
release a new version of your application. You are creating new nodes to replace
the old ones that means that you are focused on provisioning and configuration
management. You are justifying all the effort spent to implement infrastructure
as code.</p>
<p>It matters also for security because you will have a fresh container after each
update and in the case of a vulnerability or injection they will be cleaned
during the update.</p>
<p>You have also an instrument to analyse the attacked container with the command
docker diff <container_id> This command shows the differences in the file system.</container_id></p>
<p>It supports 3 events:</p>
<ul>
<li>A - Add</li>
<li>D - Delete</li>
<li>C - Change</li>
</ul>
<p>In case of attack, you can commit the attacked container to analyse it later and
replace it with the original image.</p>
<p>This flow is interesting but if you know that your application does not need to
modify the file system you can use <code class="highlighter-rouge">–read-only</code> parameters to make the fs read
only or you can share a volume with the <code class="highlighter-rouge">ro</code> suffix <code class="highlighter-rouge">-v PWD:/data:ro</code>.</p>
<p>Docker can’t fix the security issues for you, if your application can be
attacked by a code injection then you need to fix your app but Docker offers a
few utilities to make life hard for an hacker and to allow you to have more
control over your environment.</p>
<p>During this chapter we covered some practices and tools that you can follow or
use to build a safe environment.</p>
<p>In general, you need to close your application in an environment that provides
only what you need and what you know.</p>
<p>If your distribution or your container has something that you don’t have under
your control or it is unused then it is a good idea remove these dark points.</p>
<p>That’s all. Immutability is not free and it requires to keep all the tools and
processes involved in deploy, packaging up to speed because all your production
environment depends on these tools. But it’s an important piece of the puzzle.
To read more about tools like Cilium, CoreOS Clair, best practices about
registry and images you can download the pdf <a href="http://scaledocker.com">Play Safe with Docker and
Container Security</a>.</p>
Orbiter an OSS Docker Swarm Autoscaler2017-04-22T08:08:27+00:00http://gianarb.it/blog/orbiter-docker-swarm-autoscaler<iframe width="560" height="315" src="https://www.youtube.com/embed/Q1xfmfML8ok" frameborder="0" allowfullscreen=""></iframe>
<p>My presentation at the Docker HQ in San Francisco.</p>
<h2 id="autoscaling">Autoscaling</h2>
<p>One of the Cloud’s dreams is a nice world when everything magically happen. You
have unlimited resources and you are just going to use what you need and to pay
to you use.
To do what AWS provides a service called autoscaling-group for example. You can
specify some limitation and some expectation about a group of servers and AWS is
matching your expectation for you.
If you are able to make an automatic provision of a node you can use Cloudwatch
to set some alerts. When AWS trigger these alerts the austocaling-group is
creating or removing one or more instance.</p>
<h3 id="lets-try-with-an-example">let’s try with an example</h3>
<p>You have a web service and you know that for 2 hours every day you don’t need 4
EC2 because you have a lot of traffic, you need 10 of them.
You can create an autoscaling group, set some alerts:</p>
<ol>
<li>When the memory usage is more than 65% for 3 minutes start 3 new servers.</li>
<li>When the memory usage is less than 30% for 5 minutes stop 2 servers.</li>
</ol>
<p>Just to have an idea. In this way AWS knows what do you and you don’t need to
stay in front of your laptop to wait something happen. You can just do something
funny.</p>
<p>It’s something useful, if you think about a daily magazine, they usually has a
lot of traffic in the beginning of the day when all the people are usually
reading news. At that’s an easy scenario.</p>
<p>But it can also happen than a new shared on reddit or HackerNews is getting a
lot of traffic and the last thing that you are looking for is to go down just
during that spike!</p>
<h3 id="actors">Actors</h3>
<p>There are different actors in this comedy. First of all our cluster needs to be
manageable by outside via API. In this example I am going to use Docker Swarm,
Orbiter supports a basic implementation for DigitalOcean but it still requires
some toning.</p>
<p>You need to have some time series database or analytics platform that can
trigger webhook to trigger orbiter based on some metrics.</p>
<p>We ran a demo with the TICKStack (InfluxDB, Telegraf, and Kapacitor) days ago.
It’s available <a href="https://www.influxdata.com/resources/influxdata-helps-docker-auto-scale-monitoring/?ao_campid=70137000000Jgw7">at this
link</a>.</p>
<p>In the end you need to deploy <a href="https://github.com/gianarb/orbiter">orbiter</a>.</p>
<h3 id="orbiter-design-and-arch">Orbiter, design and arch</h3>
<p>Orbiter is an open source tool designed to be a cross platform autoscaler. It is
in go and it provides a REST API to handle scale requests.</p>
<p>It provides one entrypoint:</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-v</span> <span class="nt">-d</span> <span class="s1">'{"direction": true}'</span> <span class="se">\</span>
http://localhost:8000/handle/infra_scale/docker
</code></pre></div></div>
<ul>
<li><code class="highlighter-rouge">direction</code> represent how to scale your service, true means up, false means
down.</li>
<li><code class="highlighter-rouge">/handle/infra_scale/docker</code> identify the autoscaling group.
<code class="highlighter-rouge">infra_scale</code> is the autoscaler name, <code class="highlighter-rouge">docker</code> is the policy name.</li>
</ul>
<p><code class="highlighter-rouge">infra_scale</code> for example contains information about the cluster manager, where
it is, what is it? Docker or Digitalocean or what ever?</p>
<p>The policy describes how an application scale. If you know a bit Docker Swarm
<code class="highlighter-rouge">docker</code> is the name of the service.</p>
<p>Orbiter supports two different boot methods. One is via configuration:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">autoscalers</span><span class="pi">:</span>
<span class="na">infra_scale</span><span class="pi">:</span>
<span class="na">provider</span><span class="pi">:</span> <span class="s">swarm</span>
<span class="na">parameters</span><span class="pi">:</span>
<span class="na">policies</span><span class="pi">:</span>
<span class="na">docker</span><span class="pi">:</span>
<span class="na">up</span><span class="pi">:</span> <span class="s">4</span>
<span class="na">down</span><span class="pi">:</span> <span class="s">3</span>
</code></pre></div></div>
<p>The second one is actually only supported by Docker Swarm and it’s called
autodetection. In practice when you start orbiter, it’s looking for a Docker
Swarm up and running. If it finds Swarm it’s going to list all the services
deployed and it’s going to manage all the services labeled with <code class="highlighter-rouge">orbiter=true</code>.</p>
<p>By default up and down are set to 1 but you can override them with the label
orbiter.up=3 and orbiter.down=2.</p>
<p>Let’s suppose to have a Docker Swarm cluster with 3 nodes.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker node <span class="nb">ls
</span>ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS
11btq767ecqhelidu8ah1osfp <span class="k">*</span> node1 Ready Active Leader
ptre8d4bjccqi6ml6z445u0mz node2 Ready Active
q5rwi3cej9gc1vqyscwfau640 node3 Ready Active
</code></pre></div></div>
<p>I deployed a service called <a href="https://github.com/gianarb/micro">gianarb/micro</a>.
It is an open source demo application. There are different versions, I deployed
the version 1.0.0. It only shows the current IP of the container/server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker service create <span class="nt">--label</span> <span class="nv">orbiter</span><span class="o">=</span><span class="nb">true</span> <span class="se">\</span>
<span class="nt">--name</span> micro <span class="nt">--replicas</span> 3 <span class="se">\</span>
<span class="nt">-p</span> 8080:8000 gianarb/micro:1.0.0
</code></pre></div></div>
<p>You can check the number of tasks running with the command:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker service ps micro
ID NAME IMAGE NODE
DESIRED STATE CURRENT STATE ERROR
PORTS
onsqgriv3nel micro.1 gianarb/micro:1.0.0 node3
Running Running 51 seconds ago
yxtxyder7bs3 micro.2 gianarb/micro:1.0.0 node1
Running Running 51 seconds ago
lyzxxdc00052 micro.3 gianarb/micro:1.0.0 node2
Running Running 52 seconds ago
</code></pre></div></div>
<p>At this point you can visit port <code class="highlighter-rouge">8080</code> of your cluster to have a look of the
service but for this demo doesn’t really matter. We are going to start orbiter
and we are going to trigger a scaling policy to simulate a request made by our
monitoring tool.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker service create <span class="nt">--name</span> orbiter <span class="se">\</span>
<span class="nt">--mount</span> <span class="nb">type</span><span class="o">=</span><span class="nb">bind</span>,source<span class="o">=</span>/var/run/docker.sock,destination<span class="o">=</span>/var/run/docker.sock <span class="se">\</span>
<span class="nt">-p</span> 8000:8000 <span class="nt">--constraint</span> node.role<span class="o">==</span>manager <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">DOCKER_HOST</span><span class="o">=</span>unix:///var/run/docker.sock <span class="se">\</span>
gianarb/orbiter daemon <span class="nt">--debug</span>
</code></pre></div></div>
<p>I am using Docker to deploy orbiter as service. I am using the Unix Socket to
communicate with Docker Swarm and I am deploying this service into the <code class="highlighter-rouge">manager</code>
because it needs to have write permission to start and stop tasks. This can be
done only into the manager. You can configure orbiter with the variable
<code class="highlighter-rouge">DOCKER_HOST</code> to use REST API. In this way you don’t have this constraint. This
configuration in very easy to show in a demo like this one.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker service logs orbiter
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:24:56Z"</span> <span class="nv">level</span><span class="o">=</span>info
<span class="nv">msg</span><span class="o">=</span><span class="s2">"orbiter started"</span>
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:24:56Z"</span> <span class="nv">level</span><span class="o">=</span>debug
<span class="nv">msg</span><span class="o">=</span><span class="s2">"Daemon started in debug mode"</span>
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:24:56Z"</span> <span class="nv">level</span><span class="o">=</span>info
<span class="nv">msg</span><span class="o">=</span><span class="s2">"Starting in auto-detection mode."</span>
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:24:56Z"</span> <span class="nv">level</span><span class="o">=</span>info
<span class="nv">msg</span><span class="o">=</span><span class="s2">"Successfully connected to a Docker daemon"</span>
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:24:56Z"</span> <span class="nv">level</span><span class="o">=</span>debug
<span class="nv">msg</span><span class="o">=</span><span class="s2">"autodetect_swarm/micro added to orbiter. UP 1, DOWN 1"</span>
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:24:56Z"</span> <span class="nv">level</span><span class="o">=</span>info
<span class="nv">msg</span><span class="o">=</span><span class="s2">"API Server run on port :8000"</span>
</code></pre></div></div>
<p>As you can see into the logs the API are running on port 8000 and orbiter
already detected a service called <code class="highlighter-rouge">micro</code>, the one that we deployed before and
it auto-created a autoscaling group called <code class="highlighter-rouge">autodetection_swarm/micro</code>.
This is the unique name that we can use when we trigger our scale request.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>curl <span class="nt">-d</span> <span class="s1">'{"direction": true}'</span> <span class="nt">-v</span>
http://10.0.57.3:8000/handle/autodetect_swarm/micro
<span class="k">*</span> Trying 10.0.57.3...
<span class="k">*</span> TCP_NODELAY <span class="nb">set</span>
<span class="k">*</span> Connected to 10.0.57.3 <span class="o">(</span>10.0.57.3<span class="o">)</span> port 8000 <span class="o">(</span><span class="c">#0)</span>
<span class="o">&gt;</span> POST /handle/autodetect_swarm/micro HTTP/1.1
<span class="o">&gt;</span> Host: 10.0.57.3:8000
<span class="o">&gt;</span> User-Agent: curl/7.52.1
<span class="o">&gt;</span> Accept: <span class="k">*</span>/<span class="k">*</span>
<span class="o">&gt;</span> Content-Length: 19
<span class="o">&gt;</span> Content-Type: application/x-www-form-urlencoded
<span class="o">&gt;</span>
<span class="k">*</span> upload completely sent off: 19 out of 19 bytes
&lt; HTTP/1.1 200 OK
&lt; Content-Type: application/json
&lt; Date: Tue, 18 Apr 2017 09:30:35 GMT
&lt; Content-Length: 0
&lt;
<span class="k">*</span> Curl_http_done: called premature <span class="o">==</span> 0
<span class="k">*</span> Connection <span class="c">#0 to host 10.0.57.3 left intact</span>
</code></pre></div></div>
<p>With that cURL I simulated a scale request and as you can see in the log above
orbiter detected the request and it scaled up 1 task for our service called
<code class="highlighter-rouge">macro</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker service logs orbiter
orbiter.1.zop1qkwa1qxy@node1 | POST /handle/autodetect_swarm/micro HTTP/1.1
orbiter.1.zop1qkwa1qxy@node1 | Host: 10.0.57.3:8000
orbiter.1.zop1qkwa1qxy@node1 | Accept: <span class="k">*</span>/<span class="k">*</span>
orbiter.1.zop1qkwa1qxy@node1 | Content-Length: 19
orbiter.1.zop1qkwa1qxy@node1 | Content-Type:
application/x-www-form-urlencoded
orbiter.1.zop1qkwa1qxy@node1 | User-Agent: curl/7.52.1
orbiter.1.zop1qkwa1qxy@node1 |
orbiter.1.zop1qkwa1qxy@node1 | <span class="o">{</span><span class="s2">"direction"</span>: <span class="nb">true</span><span class="o">}</span>
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:30:35Z"</span> <span class="nv">level</span><span class="o">=</span>info
<span class="nv">msg</span><span class="o">=</span><span class="s2">"Received a new request to scale up micro with 1 task."</span> direc
<span class="nv">tion</span><span class="o">=</span><span class="nb">true </span><span class="nv">service</span><span class="o">=</span>micro
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:30:35Z"</span> <span class="nv">level</span><span class="o">=</span>debug
<span class="nv">msg</span><span class="o">=</span><span class="s2">"Service micro scaled from 3 to 4"</span> <span class="nv">provider</span><span class="o">=</span>swarm
orbiter.1.zop1qkwa1qxy@node1 | <span class="nb">time</span><span class="o">=</span><span class="s2">"2017-04-18T09:30:35Z"</span> <span class="nv">level</span><span class="o">=</span>info
<span class="nv">msg</span><span class="o">=</span><span class="s2">"Service micro scaled up."</span> <span class="nv">direction</span><span class="o">=</span><span class="nb">true </span><span class="nv">service</span><span class="o">=</span>micro
</code></pre></div></div>
<p>We can verify the current number of tasks that are running for <code class="highlighter-rouge">micro</code> and we
can see that it’s not 3 as before but 4.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker service <span class="nb">ls
</span>ID NAME MODE REPLICAS
IMAGE
azi8zyeor5eb micro replicated 4/4
gianarb/micro:1.0.0
ezklgb6uak8b orbiter replicated 1/1
gianarb/orbiter:latest
</code></pre></div></div>
<p>This project is open source on
<a href="https://github.com/gianarb/orbiter">github.com/gianarb/orbiter</a> you can have a look on
it, try and leave some feedback or request if you need something different.</p>
<p>PR are also open if you are working with a different cluster manager or with a
different provider, add a new one is very easy. It’s just a new interface to
implement.</p>
LinuxKit operating system built for container2017-04-18T10:08:27+00:00http://gianarb.it/blog/linuxkit-operating-system-build-for-containers<p>Linuxkit is a new project presented by Docker during the DockerCon 2017. If we
look at the description of the project on
<a href="https://github.com/linuxkit/linuxkit">GitHub</a>:</p>
<blockquote>
<p>A secure, portable and lean operating system built for containers</p>
</blockquote>
<p>I am feeling already exited. I was an observer of the project when <a href="https://twitter.com/justincormack">Justin
Cormack</a> and the other
<a href="https://github.com/linuxkit/linuxkit/graphs/contributors">contributors</a> was
working on a private repository. I was invited as part of ci-wg group into the
CNCF and I loved this project from the first day.</p>
<p>You can think about linuxkit as a builder for Linux operating system everything
based on containers.</p>
<p>It’s a project that can stay behind your continuous integration system to allow
us to test on different kernel version and distribution. You can a light kernels
with all the services that you need and you can create different outputs
runnable on cloud providers as Google Cloud Platform, with Docker or with QEMU.</p>
<h2 id="continuous-delivery-new-model">Continuous delivery, new model</h2>
<p>I am not really confident about Google Cloud Platform but just to move over I am
going to do some math with AWS as provider.
Let’s suppose that I have the most common continuous integration system, one big
box always up an running configured to support all your projects or if you are
already good you are running containers to have separated and isolated
environment.</p>
<p>Let’s suppose that you Jenkins is running all times on m3.xlarge:</p>
<p><code class="highlighter-rouge">m3.xlarge</code> used 100% every months costs 194.72$.</p>
<p>Let’s have a dream. You have a very small server with just a frontend
application for your CI and all jobs are running in a separate instance, tiny as
a t2.small.</p>
<p><code class="highlighter-rouge">t2.small</code> used only 1 hour costs 0.72$ .</p>
<p>I calculated 1 hour because it’s the minimum that you can pay and I hope that
your CI job can run for less than 1 hour.
Easy math to calculate the number of builds that you need to run to pay as you
was paying before.</p>
<p>194.72 / 0.72 ~ 270 builds every month.</p>
<p>If you are running less than 270 builds a months you can save some money
too. But you have other benefits:</p>
<ol>
<li>More jobs, more instances. Very easy to scale. Easier that Jenkins
master/slave and so on.</li>
<li>How many times during holidays your Jenkins is still up and running without
to have nothing to do? During these days you are just paying for the frontend
app.</li>
</ol>
<p>And these are just the benefit to have a different setup for your continuous
delivery.</p>
<h2 id="linuxkit-ci-implementation">LinuxKit CI implementation</h2>
<p>There is a directory called
<a href="https://github.com/linuxkit/linuxkit/tree/master/test">./test</a> that contains
some linuxkit use case but I am going to explain in practice how linuxkit is
tested. Because it uses itself, awesome!</p>
<p>In first you need to download and compile linuxkit:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone github.com:linuxkit/linuxkit <span class="nv">$GOPATH</span>/src/github.com/linuxkit/linuxkit
make
./bin/moby
</code></pre></div></div>
<p>You can move it in your <code class="highlighter-rouge">$PATH</code> with <code class="highlighter-rouge">make install</code>.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ moby
Please specify a command.
USAGE: moby [options] COMMAND
Commands:
build Build a Moby image from a YAML file
run Run a Moby image on a local hypervisor or remote cloud
version Print version information
help Print this message
Run 'moby COMMAND --help' for more information on the command
Options:
-q Quiet execution
-v Verbose execution
</code></pre></div></div>
<p>At the moment the CLI is very simple, the most important commands are build and
run. linuxkit is based on YAML file that you can use to describe your kernel,
with all applications and all the services that you need. Let’s start with the
<a href="https://github.com/linuxkit/linuxkit/blob/master/test/test.yml">linuxkit/test/test.yml</a>.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">kernel</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mobylinux/kernel:4.9.x"</span>
<span class="na">cmdline</span><span class="pi">:</span> <span class="s2">"</span><span class="s">console=ttyS0"</span>
<span class="na">init</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">mobylinux/init:8375addb923b8b88b2209740309c92aa5f2a4f9d</span>
<span class="pi">-</span> <span class="s">mobylinux/runc:b0fb122e10dbb7e4e45115177a61a3f8d68c19a9</span>
<span class="pi">-</span> <span class="s">mobylinux/containerd:18eaf72f3f4f9a9f29ca1951f66df701f873060b</span>
<span class="pi">-</span> <span class="s">mobylinux/ca-certificates:eabc5a6e59f05aa91529d80e9a595b85b046f935</span>
<span class="na">onboot</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">dhcpcd</span>
<span class="err"> </span><span class="na">image</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mobylinux/dhcpcd:0d4012269cb142972fed8542fbdc3ff5a7b695cd"</span>
<span class="err"> </span><span class="na">binds</span><span class="pi">:</span>
<span class="err"> </span> <span class="pi">-</span> <span class="s">/var:/var</span>
<span class="err"> </span> <span class="s">- /tmp:/etc</span>
<span class="err"> </span><span class="na">capabilities</span><span class="pi">:</span>
<span class="err"> </span> <span class="pi">-</span> <span class="s">CAP_NET_ADMIN</span>
<span class="err"> </span> <span class="s">- CAP_NET_BIND_SERVICE</span>
<span class="err"> </span> <span class="s">- CAP_NET_RAW</span>
<span class="err"> </span><span class="na">net</span><span class="pi">:</span> <span class="s">host</span>
<span class="err"> </span><span class="na">command</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">/sbin/dhcpcd"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">--nobackground"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">-f"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">/dhcpcd.conf"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">-1"</span><span class="pi">]</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">check</span>
<span class="err"> </span><span class="na">image</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mobylinux/check:c9e41ab96b3ea6a3ced97634751e20d12a5bf52f"</span>
<span class="err"> </span><span class="na">pid</span><span class="pi">:</span> <span class="s">host</span>
<span class="err"> </span><span class="na">capabilities</span><span class="pi">:</span>
<span class="err"> </span> <span class="pi">-</span> <span class="s">CAP_SYS_BOOT</span>
<span class="err"> </span><span class="na">readonly</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">outputs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">format</span><span class="pi">:</span> <span class="s">kernel+initrd</span>
<span class="pi">-</span> <span class="na">format</span><span class="pi">:</span> <span class="s">iso-bios</span>
<span class="pi">-</span> <span class="na">format</span><span class="pi">:</span> <span class="s">iso-efi</span>
<span class="pi">-</span> <span class="na">format</span><span class="pi">:</span> <span class="s">gcp-img</span>
</code></pre></div></div>
<p>Linuxkit builds everythings inside a container, it means that you don’t need a
lot of dependencies it’s very easy to use. It generates different <code class="highlighter-rouge">output</code> in
this case <code class="highlighter-rouge">kernel+initrd</code>, <code class="highlighter-rouge">iso-bios</code>, <code class="highlighter-rouge">iso-efi</code>, <code class="highlighter-rouge">gpc-img</code> depends of the
platform that you are interested to use to run your kernel.</p>
<p>I am trying to explain a bit how this YAML works. You can see that there are
different primary section: <code class="highlighter-rouge">kernel</code>, <code class="highlighter-rouge">init</code>, <code class="highlighter-rouge">onboot</code>, <code class="highlighter-rouge">service</code> and so on.</p>
<p>Pretty much all of them contains the keyword <code class="highlighter-rouge">image</code> because as I said before
everything is applied on containers, in this example they are store in
<a href="https://hub.docker.com/u/mobylinux/">hub.docker.com/u/mobylinux/</a>.</p>
<p>The based kernel is <code class="highlighter-rouge">mobylinux/kernel:4.9.x</code>, I am just reporting what the
<a href="https://github.com/linuxkit/linuxkit#yaml-specification">README.md</a> said:</p>
<ul>
<li><code class="highlighter-rouge">kernel</code> specifies a kernel Docker image, containing a kernel and a
filesystem tarball, eg containing modules. The example kernels are built from
<code class="highlighter-rouge">kernel/</code></li>
<li><code class="highlighter-rouge">init</code> is the base <code class="highlighter-rouge">init</code> process Docker image, which is unpacked as the base
system, containing <code class="highlighter-rouge">init</code>, <code class="highlighter-rouge">containerd</code>, <code class="highlighter-rouge">runc</code> and a few tools. Built from
<code class="highlighter-rouge">pkg/init/</code></li>
<li><code class="highlighter-rouge">onboot</code> are the system containers, executed sequentially in order. They
should terminate quickly when done.</li>
<li><code class="highlighter-rouge">services</code> is the system services, which normally run for the whole time the
system is up</li>
<li><code class="highlighter-rouge">files</code> are additional files to add to the image</li>
<li><code class="highlighter-rouge">outputs</code> are descriptions of what to build, such as ISOs.</li>
</ul>
<p>At this point we can try it. If you are on MacOS as I was you don’t need to
install anything one of the runner supported by <code class="highlighter-rouge">linuxkit</code> is <code class="highlighter-rouge">hyperkit</code> it
means that everything is available in your system.</p>
<p><code class="highlighter-rouge">./test</code> contains different test suite but now we will stay focused on
<code class="highlighter-rouge">./test/check</code> directory. It contains a set of checks to validate how the
kernel went build by LinuxKit. They are the smoke tests that are running on each
new pull request created on the repository for example.</p>
<p>As I said everything runs inside a container, if you look into the check
directory there is a makefile that build a mobylinux/check image, that image
went run in LinuxKit, into the <code class="highlighter-rouge">test.yml</code> file:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">onboot</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">check</span>
<span class="err"> </span><span class="na">image</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mobylinux/check:c9e41ab96b3ea6a3ced97634751e20d12a5bf52f"</span>
<span class="err"> </span><span class="na">pid</span><span class="pi">:</span> <span class="s">host</span>
<span class="err"> </span><span class="na">capabilities</span><span class="pi">:</span>
<span class="err"> </span> <span class="pi">-</span> <span class="s">CAP_SYS_BOOT</span>
<span class="err"> </span><span class="na">readonly</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div></div>
<p>You can use the
<a href="https://github.com/linuxkit/linuxkit/blob/master/test/check/Makefile">Makefile</a>
inside the check directory to build a new version of check, you can just use
the command <code class="highlighter-rouge">make</code>.</p>
<p>When you have the right version of your test we can build the image used by moby:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd $GOPATH/src/github.com/linuxkit/linuxkit
moby build test/test.yml
</code></pre></div></div>
<p>Part of the output is:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Create outputs:
test-bzImage test-initrd.img test-cmdline
test.iso
test-efi.iso
test.img.tar.gz
</code></pre></div></div>
<p>And if you look into the directory you can see that there are all these files
into the root. These files can be run from qemu, google cloud platform,
hyperkit and so on.</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>moby run <span class="nb">test</span>
</code></pre></div></div>
<p>On MacOS with this command LinuxKit is using hyperkit to start a VM, I can not copy
paste all the output but you can see the hypervisor logs:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>virtio-net-vpnkit: initialising, opts="path=/Users/gianlucaarbezzano/Library/Containers/com.docker.docker/Data/s50"
virtio-net-vpnkit: magic=VMN3T version=1 commit=0123456789012345678901234567890123456789
Connection established with MAC=02:50:00:00:00:04 and MTU 1500
early console in extract_kernel
input_data: 0x0000000001f2c3b4
input_len: 0x000000000067b1e5
output: 0x0000000001000000
output_len: 0x0000000001595280
kernel_total_size: 0x000000000118a000
booted via startup_32()
Physical KASLR using RDRAND RDTSC...
Virtual KASLR using RDRAND RDTSC...
Decompressing Linux... Parsing ELF... Performing relocations... done.
Booting the kernel.
[ 0.000000] Linux version 4.9.21-moby (root@84baa8e89c00) (gcc version 6.2.1 20160822 (Alpine 6.2.1) ) #1 SMP Sun Apr 9 22:21:32 UTC 2017
[ 0.000000] Command line: earlyprintk=serial console=ttyS0
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[ 0.000000] x86/fpu: Using 'eager' FPU context switches.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
</code></pre></div></div>
<p>When the VM is ready LinuxKit is starting all the <code class="highlighter-rouge">init</code>, <code class="highlighter-rouge">onboot</code>, the logs is
easy to understand as the <code class="highlighter-rouge">test.yml</code> is starting <code class="highlighter-rouge">containerd</code>, <code class="highlighter-rouge">runc</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>init:
- mobylinux/init:8375addb923b8b88b2209740309c92aa5f2a4f9d
- mobylinux/runc:b0fb122e10dbb7e4e45115177a61a3f8d68c19a9
- mobylinux/containerd:18eaf72f3f4f9a9f29ca1951f66df701f873060b
- mobylinux/ca-certificates:eabc5a6e59f05aa91529d80e9a595b85b046f935
onboot:
- name: dhcpcd
image: "mobylinux/dhcpcd:0d4012269cb142972fed8542fbdc3ff5a7b695cd"
binds:
- /var:/var
- /tmp:/etc
capabilities:
- CAP_NET_ADMIN
- CAP_NET_BIND_SERVICE
- CAP_NET_RAW
net: host
command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
- name: check
image: "mobylinux/check:c9e41ab96b3ea6a3ced97634751e20d12a5bf52f"
pid: host
capabilities:
- CAP_SYS_BOOT
readonly: true
</code></pre></div></div>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Welcome to LinuxKit
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
/ # INFO[0000] starting containerd boot... module=containerd
INFO[0000] starting debug API... debug="/run/containerd/debug.sock" module=containerd
INFO[0000] loading monitor plugin "cgroups"... module=containerd
INFO[0000] loading runtime plugin "linux"... module=containerd
INFO[0000] loading snapshot plugin "snapshot-overlay"... module=containerd
INFO[0000] loading grpc service plugin "healthcheck-grpc"... module=containerd
INFO[0000] loading grpc service plugin "images-grpc"... module=containerd
INFO[0000] loading grpc service plugin "metrics-grpc"... module=containerd
</code></pre></div></div>
<p>The last step is the <code class="highlighter-rouge">check</code> that runs the real test suite:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kernel config test succeeded!
info: reading kernel config from /proc/config.gz ...
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
........
.......
Moby test suite PASSED
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
[ 3.578681] ACPI: Preparing to enter system sleep state S5
[ 3.579063] reboot: Power down
</code></pre></div></div>
<p>The last log is the output of
<a href="https://github.com/linuxkit/linuxkit/blob/master/test/check/check-kernel-config.sh">check-kernel-config.sh</a>
files.</p>
<p>If you are on linux you can do the same command but by the default you are going
to use <a href="http://www.qemu-project.org/">qemu</a> an open source machine emulator.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get install qemu
</code></pre></div></div>
<p>I did some test in my Asus Zenbook with Ubuntu, when you run <code class="highlighter-rouge">moby run</code> this is
the command executed with qemu:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/usr/bin/qemu-system-x86_64 -device virtio-rng-pci -smp 1 -m 1024 -enable-kvm
-machine q35,accel=kvm:tcg -kernel test-bzImage -initrd test-initrd.img -append
console=ttyS0 -nographic
</code></pre></div></div>
<p>By default is testing on <code class="highlighter-rouge">x86_64</code> but qemu supports a lot of other archs and
devices. You can simulate an arm and a rasperry pi for example. At the
moment LinuxKit is not ready to emulate other architecture. But this is the main
scope for this project. It’s just a problem of time. It will be able soon!</p>
<p>Detect if the build succeed or failed is not easy as you probably expect. The
status inside the VM is not the one that you get in your laptop. At the moment
to understand if the code in your PR is good or bad we are parsing the output:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>define check_test_log
@cat $1 |grep -q 'Moby test suite PASSED'
endef
</code></pre></div></div>
<p><a href="https://github.com/linuxkit/linuxkit/blob/master/Makefile">./linuxkit/Makefile</a></p>
<p>Explain how linuxkit tests itself at the moment is the best way to get how it
works. It is just one piece of the puzzle, if you have a look here <a href="https://github.com/linuxkit/linuxkit/pulls">every
pr</a> has a GitHub Status that point to
a website that contains logs related that particular build. That part is not
managed by linuxkit because it’s only the builder used to create the
environment. All the rest is managed by
<a href="https://github.com/docker/datakit">datakit</a>. I will speak about it probably in
another blogpost.</p>
<h2 id="conclusion">Conclusion</h2>
<p>runc, docker, containerd, rkt but also Prometheus, InfluxDB, Telegraf a lot of
projects supports different architecture and they need to run on different
kernels with different configuration and capabilities. They need to run on your
laptop, in your IBM server and in a Raspberry Pi.</p>
<p>This project is in an early state but I understand why Docker needs something
similar and also, other projects as I said are probably going to get some
benefits from a solution like this one. Have it open source it’s very good and
I am honored to be part of the amazing group that put this together. I just did
some final tests and I tried to understand how it’s designed and how it works.
This is the result of my test. I hope that can be helpful to start in the right
mindset.</p>
<p>My plan is to create a configuration to test InfluxDB and play a bit with <code class="highlighter-rouge">qemu</code>
to test it on different architectures and devices. Stay around a blogpost will
come!</p>
<p>Some Links:</p>
<ul>
<li><a href="https://blog.docker.com/2017/04/introducing-the-moby-project/">INTRODUCING MOBY PROJECT: A NEW OPEN-SOURCE PROJECT TO ADVANCE THE SOFTWARE
CONTAINERIZATION MOVEMENT</a></li>
<li><a href="https://github.com/linuxkit">github.com/linuxkit</a></li>
<li><a href="https://github.com/moby">github.com/moby</a></li>
</ul>
<p class="text-muted">
Reviewers: <a href="https://twitter.com/justincormack">Justin Cormack</a>
</p>
<div class="post row">
<div class="col-md-12">
<div class="bs-callout bs-callout-info row">
<div class="row">
<div class="col-md-12">
<h2><a href="//gianarb.it/blog/docker-the-fundamentals" target="_blank">get "Docker the Fundamentals"</a> <small>by. Drive your boat as a Captain</small></h2>
</div>
</div>
<div class="row">
<div class="col-md-3">
<a href="//gianarb.it/blog/docker-the-fundamentals" target="_blank"><img src="/img/the-fundamentals.jpg" class="img-fluid" /></a>
</div>
<div class="col-md-9">
<p>
You can get the Chapter 2 of the book <a href="http://scaledocker.com" target="_blank">"Drive your boat as a Captain"</a> just leave click on the
cover and leave your email to receive a free copy.</p>
<p>This chapter is getting started with Docker Engine and the basic
concept around registry, pull, push and so on. It's a good way to start from
zero with Docker.</p>
</div>
</div>
</div>
</div>
</div>
Containers why we are here2017-03-12T08:08:27+00:00http://gianarb.it/blog/containers-why-we-are-here<blockquote>
<p>“It is change, continuing change, inevitable change, that is the dominant
factor in society today. No sensible decision can be made any longer without
taking into account not only the world as it is, but the world as it will be…
This, in turn, means that our statesmen, our businessmen, our everyman must take
on a science fictional way of thinking” Asimov, 1981</p>
</blockquote>
<h1 id="isolation-and-virtualization">Isolation and Virtualization</h1>
<p>I can see clearly two kind of invention: the ones that allow people to do
something they couldn’t do before and the ones that let them do something
better. Fire, for example, gave people the chance to cook food, push away wild
beasts and warm themselves up during cold nights. Many years later, electricity
let people warm their houses just by pushing a button. After wheels discovery
people began to travel and to trades goods, but was only with car’s invention
that they might do it faster and efficiently. Similarly, the web creates a huge
network, able to connect people all over the world, web application gave people
tools to use and customise such a complex system. Under this perspective,
container is one of the main revolution of the last years, a unique tool that
helps with app management and development. Let’s discover something more about
the real story of containers.</p>
<p>We have not a lot of documentation about why Bill Joy 18th March 1982 added
chroot into the BSD probably to emulate him solutions and program is an isolated
root. That’s was amazing but not enough few years later in 1991 Bill Cheswick
extended chroot with security features provided by FreeBSD and implemented the
“jails” and in 2000 he introduced what we know as the proper jails command now
our chroots can not be anything, anywhere out of themself. When you start a
process in a chroot the PID is one and there is only that process but from
outside you can see all processes that are running in a chroot. Our
applications can not stay in a jail! They need to communicate with outside,
exchange information and so on. To solve this problem in 2002 in the kernel
version 2.4.19 a group of developers like Eric W. Biederman, Pavel Emelyanov
introduced the namespace feature to manage system resources like network,
process and file system.</p>
<p>This is just a bit of history about how the ecosystem spin up, in the end of
this chapter we will try to understand how why Docker arrives on the scene, but
the main goal of this book is on another layer and on another complexity, we are
here to understand how manage all this things in cloud and how to design a
distributed system but you know the past is important to build a solid future.</p>
<p>All this great features are now popular under the name of container, nothing
really news and this is one of the reason about why all this things are amazing!
They are under the hood from a while! Solid and tested feature put together and
made usable.</p>
<p>Nothing to say about the importance for a system to being isolated: isolation
helps us to usefully manage resources, security and monitoring, in the best way,
false problems creation in specific applications, often not even related to our
app.</p>
<p>The most common solution is virtualization: you can use an hypervisor to create
virtual server in a single machine. There are different kind of virtualization:</p>
<ul>
<li>Full virtualization</li>
<li>Para virtualization like Virtual Machine, Xen, VMware</li>
<li>Operating System virtualization like Containers</li>
<li>Application virtualization like JVM.</li>
</ul>
<p><img class="img-fluid" src="/img/virtualization.png" />
<a href="https://fntlnz.wtf/post/why-containers/" target="_blank"><small>img from fntlnz’s blog. Thanks</small></a></p>
<p>The main differences between them is how they abstract the layers, application,
processing, network, storage and also about how the superior level interact with
underlying level. For example into the Full virtualization the hardware is
virtualized, into the para virtualization not.</p>
<p>Container is an operation-system-level virtualization. The main difference
between Container and Virtual Machine is the layer: the first works on the
operating system, the second on the hardware layer.</p>
<p>When we speak about container we are focused on the application virtualization
and on a specific feature provided by the kernel called Linux Containers (LXC):
what we do when we build containers is create new isolated Linux systems into
the same host, it means that we can not change the operation system for example
because our virtualization layer doesn’t allow us to run Linux containers out of
Linux.</p>
<h1 id="the-reasons">The reasons</h1>
<p>Revolutions are not related to a single and specific event but come from
multiple movements and changes: Container is just a piece of the story.</p>
<p>Cloud Computing allowed us to think about our infrastructure as an instable
number of servers that can scale up and down, in a reasonable short amount of
time, with less money and without the investment requested to manage a big
infrastructure made of more than one datacenter across the world.</p>
<p>As a consequence, applications that had been in a cellar, now are on Amazon Web
Service, with a load balancer and maybe different availability zone. This
allowed little teams and medium companies, without datacenter and
infrastructures, to think about concept like distribution, high availability,
redundancy. Evolution never stop .</p>
<p>Once our applications are running in few virtual machines, our business will
grow up so we start to scale up and down this servers to serve all our users.
We experimented few benefits but also a lot of issues related, for example, to
the time requested for managing this dynamism; moreover big applications are
usually more expensive to scale.</p>
<p>Our application can only grow but the deploy can be really expensive. We
discovered that the behavior of an application is not the same across all of our
services and entrypoint, because few of them receive more traffic that others.
So, we started to split our big applications in order to make them easy to scale
and monitor. The problem was that, in order to maintain our standard, we need to
find a way to keep them isolated, safe and able to communicate each others.</p>
<p>The Microservices Architecture arrived and companies like Netflix, Amazon,
Google and others counts hundreds and hundreds of little and specific of
services that together work to serve big and profitable products. Netlix is one
of first companies that started sharing the way they build Netlix.com: with more
that 400 microservices, they managed feature like registration, streaming,
rankins and all what the application provides. At the moment, Containers are
the best solution for managing a dense and dynamic environment with a good
control, security and for moving your application between servers.</p>
<p class="text-muted">
Reviewers: Arianna Scarcella, <a href="https://twitter.com/TheBurce">Jenny Burcio</a>
</p>
About your images, security tips2016-12-28T08:08:27+00:00http://gianarb.it/blog/about-your-images-security-tips<p>Everything unnecessary in your system could be a very stupid vulnerability. We
already spoke about this idea in the capability chapter and the same rule
exists when we build an image. Having tiny images with only what our
application needs to run is not just a goal in terms of distribution but also
in terms of cost of maintenance and security. If you have some small
experience with docker already you probably know the
<a href="https://hub.docker.com/_/alpine/">alpine</a> image. It is build
from the Alpine distribution and it’s only 5MB size, if your application can
run inside it then this is a very good optimization that you can do. What
about your binaries? Can your application run standalone? If the answer is yes
you can think about a very very minimal image. scratch is usually used as a
base for other images like debian and ubuntu but you can also use it to run
your golang binary and let me show you something with our micro application.
In the <a href="https://github.com/gianarb/micro/releases/tag/1.0.0">release page</a>,
there are a list of binaries already compiled and ready to be used. In this
case we can download the linux_386 binary.</p>
<p><img class="img-fluid" src="/img/security-image/micro-release.png" /></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-SsL</span> https://github.com/gianarb/micro/releases/download/1.0.0/micro_1.0.0_linux_386 <span class="o">&gt;</span> micro
</code></pre></div></div>
<p>And we know we can include this binary in the scratch image with this Dockerfile</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM scratch
ADD ./micro /micro
EXPOSE 8000
CMD <span class="o">[</span><span class="s2">"/micro"</span><span class="o">]</span>
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker build <span class="nt">-t</span> micro-scratch <span class="nb">.</span>
docker run <span class="nt">-p</span> 8000:8000 micro-scratch
</code></pre></div></div>
<p>The expectation is an http application on port 8000 but the main difference is
the size of the image, the old one from alpine is 12M the new one is 5M.</p>
<p>The scratch image is impossibile to use with all applications but if you have a
binary you can remove a lot of unused overhead.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>Another way to understand the status of your image is to scan it to detect
security vulnerabilities or exposures. Docker Hub and Docker Cloud can do it
for private images. This is a great feature to have in your pipeline to scan
an image after a build.</p>
<p>CoreOS provides an open source project called <a href="https://github.com/coreos/clair">clair</a> to do the same in your environment.</p>
<p>It is an application in Golang that exposes a set of HTTP API to
pull, push and analyse images. It downloads vulnerabilities from different
sources like <a href="https://security-tracker.debian.org/tracker">Debian Security
Tracker</a> or <a href="https://www.redhat.com/security/data/metrics/">RedHat Security
Data</a>. Each vulnerability is
stored in Postgres. Clair works like static analyzer, this means that it
doesn’t need to run our container to scan it but it persists different checks
directly into the filesystem of the image.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker run <span class="nt">-it</span> <span class="nt">-p</span> 5000:5000 registry
</code></pre></div></div>
<p>With this command we are running a private registry to use as a source for the
image to scan</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker pull gianarb/micro:1.0.0
docker tag gianarb/micro:1.0.0 localhost:5000/gianarb/micro:1.0.0
docker push localhost:5000/gianarb/micro:1.0.0
</code></pre></div></div>
<p>Now that we pushed in our private repo the micro image we can setup clair.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkdir <span class="nv">$HOME</span>/clair-test/clair_config
<span class="nb">cd</span> <span class="nv">$HOME</span>/clair-test
curl <span class="nt">-L</span> https://raw.githubusercontent.com/coreos/clair/v1.2.2/config.example.yaml <span class="nt">-o</span> clair_config/config.yaml
curl <span class="nt">-L</span> https://raw.githubusercontent.com/coreos/clair/v1.2.2/docker-compose.yml <span class="nt">-o</span> docker-compose.yml
</code></pre></div></div>
<p>Modify <code class="highlighter-rouge">$HOME/clair_config/config.yml</code> and add the proper source
<code class="highlighter-rouge">postgresql://postgres:password@postgres:5432?sslmode=disable</code></p>
<p>Now you can run the following command to start postgres and clair:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker-compose up
</code></pre></div></div>
<p>To make our test easier, we will use another CLI called hyperclair that is just
a client to work with this application. If you are using Mac OS, you can follow
the above commands, if you are in another OS you can find the correct url in
the release page</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-SSl</span> https://github.com/wemanity-belgium/hyperclair/releases/download/0.5.2/hyperclair-darwin-386 <span class="o">&gt;</span> ~/hyperclair
chmod 755 ~/hyperclair
</code></pre></div></div>
<p>Now we have an executable in ~/hyperclair</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~/hyperclair pull localhost:5000/gianarb/micro:1.0.0
~/hyperclair push localhost:5000/gianarb/micro:1.0.0
~/hyperclair analyze localhost:5000/gianarb/micro:1.0.0
~/hyperclair report localhost:5000/gianarb/micro:1.0.0
</code></pre></div></div>
<p>The generated report looks like this:</p>
<p><img class="img-fluid" src="/img/security-image/report-clair.png" /></p>
<p>Hyperclair is just one of the implementations of clair, you can decide to use
it or build your own implementation in your pipeline.</p>
<div class="post row">
<div class="col-md-12">
<div class="bs-callout bs-callout-info row">
<div class="row">
<div class="col-md-12">
<h2>Keep in touch with "Container Security"<small>by. <a href="http://scaledocker.com" target="_blank">Drive your boat as a Captain</a></small></h2>
</div>
</div>
<div class="row">
<div class="col-md-3">
<img src="/img/container-security.png" class="img-fluid" />
</div>
<div class="col-md-9">
<p>"Container Security" is a Chapter of "Driver your boat as a
Captain". Leave your email to receive other articles related Container Security.</p>
<!-- Begin MailChimp Signup Form -->
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=35a9b9f10e" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL"></label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_35a9b9f10e" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
</div>
</div>
</div>
</div>
</div>
Docker registry to ship and manage your containers.2016-12-14T08:08:27+00:00http://gianarb.it/blog/docker-registry-to-ship-your-containers<p>Build and Run containers is important but ship them out of your laptop is the
best part! The Registry is a very important tool that requires a bit more
attention. A Registry is used to store and manage your images and all your
layers. You can use a storage to upload and download them across your servers
and to share them with your colleagues.</p>
<p>The most popular one is hub.docker.com
it contains different kind of images: public, official and private. You can
create an account and push your images or build them for example from a github
or bitbucket repository. The integration with GitHub and Bitbucket is called
“Automated Builds”. it allows you to create a continuous integration
environment for your images, when you select “Create” and “Automated Builds”
you can specify a repository and a path of your Dockerfile. You can specify
more that one path from the same repository to build more that one image tag.
In this way you can centralize and build your images every time that a new
change is pushed into the repository. It also supports organizations to split
your images in different groups and manage visibility of them in case of
private images.</p>
<p>By default any developer can push their image to registry and
they’ll be public and free for other developers to use. Official images are
those public images selected and maintained from specific organization or
member of the communities, the idea is that they have a better quality or who
provides them are usually involved into the product development. A set of
official images are: Nginx, Redis, MySql, PHP, Go and so on
<a href="https://hub.docker.com/explore">https://hub.docker.com/explore</a>.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>Docker Hub offers different plan to store
private images, all people has one for free but if you need more you can pay a
plan and store more.</p>
<p>Registry is not just a tool but it’s a specification, it
describe how expose capabilities has pull, push, search and so on. This
solution allowed the ecosystem to implement these rules in other projects and
save the compatibility with the Docker Client and with the other runtime engine
that use this capability. It’s for this reason that other providers as
Kubernetes, Cloud Foundry supports download from Docker Hub. This specification
has 2 version, v1 and v2 the most famous registries implement both standard and
they fallback from v2 to v1 for features that are not supported yet. For
example Search is not supported at the moment into the v2 but only in v1.</p>
<p>If you are looking for an In-House solution you have different tools available
online. The first one is distribution. It is provided by Docker, it’s open
source and offers a very tiny registry that you can start and store in your
server. It also supports different storage like the local filesystem and S3.
This feature is very interesting because usually the size of the images and the
number of layers increase very fast and you also need to keep them safe with
backup and redundancy policies for high availability. This is very important if
your environment is based on containers it means that your register is a core
part of your company. Let’s start a Docker Distribution:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-d</span> <span class="nt">-p</span> 5000:5000 <span class="nt">--name</span> registry registry:2
</code></pre></div></div>
<p>In docker the default registry is hub.docker.com it means that when we push or
pull an image we are reaching this registry:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker pull alpine
</code></pre></div></div>
<p>To push our images in another registry we need to tag them:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker tag alpine 127.0.0.1:5000/alpine
</code></pre></div></div>
<p>With this command you tagged the alpine to a registry 127.0.0.1:5000 because as
we said in previous chapters the name of the image contains a lot of
information:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>REGISTRY/NAME:VERSION
</code></pre></div></div>
<p>The default registry is hub.docker.com a name could be simple as alpine or with
a username matt/alpine and you can pin a specific build with a version you can
use semver or for example the sha of the commit the default VERSION is latest.</p>
<p>Now that we have a new tag we can push and pull it in from our registry:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker push 127.0.0.1:5000/alpine
<span class="nv">$ </span>docker pull 127.0.0.1:5000/alpine
</code></pre></div></div>
<p>A very important information to remember when you start a customer registry is
that every layers, every build is stored and it’s very easy to have a big
registry, you need to monitor the instance to be sure that your server has
enough disks space and also take care about high availability. In a real
environment the registry it the core of your infrastructure, developers use it
to pull and push build and also to put a version in production. Take care of
your registry.</p>
<p>Other that Docker provided registry there are few alternatives. <a href="https://www.sonatype.com/nexus-repository-sonatype">Nexus</a> is a registry manager that
support a lot of languages and packages if you are a Java developer you know
it. Nexus supports Docker Registry API v1 and v2. The Docker registry
specification is young but it has 2 version already.</p>
<p>We can use the image provided by Sonatype and start our Nexus repository:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-d</span> <span class="nt">-p</span> 8082:8082 <span class="nt">-p</span> 8081:8081 <span class="se">\</span>
<span class="nt">-v</span> /tmp/sonata:/sonatype-work <span class="nt">--name</span> nexus sonatype/nexus3
<span class="nv">$ </span>docker logs <span class="nt">-f</span> nexus
</code></pre></div></div>
<p>When our log tells us that Nexus is ready we can reach the ui from our browser
http://localhost:8081/ or with the IP of your Docker Machine if you are using
Docker for Mac/Windows or Docker in Linux. The default credentials are username
admin and password admin123.</p>
<p><img class="img-fluid" src="/img/docker-registry/nexus-image-loaded.png" /></p>
<p>First of all we need to create a new Hosted Repository for Docker, we need to
press the Settings Icon top left of the page, Repositories and Create
Repository. I called mine mydocker and you need to specify an HTTP port for
that repository, we shared port 8082 during the run and for this reason I chose
8082.</p>
<p><img class="img-fluid" src="/img/docker-registry/nexus-create-repo.png" /></p>
<p>Nexus has different kind of repositories Host means that it’s self hosted but
you can also create a Proxy Repository to proxy for example the official Docker
Hub.
Now we need to login to out docker registry:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker login 127.0.0.1:8082
</code></pre></div></div>
<p>Now we can tag an alpine and push the tag into the repository</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker tag alpine 127.0.0.1:8082/alpine
<span class="nv">$ </span>docker push 127.0.0.1:8082/alpine
</code></pre></div></div>
<p>You can go in Assets click on mydocker repository and see that your image is
correctly stored.</p>
<p><a href="https://about.gitlab.com/">GitLab</a> has also a container registry. GitLab uses
it to manage build and it’s available for you from version 8.8 if you are
already using this tool.</p>
<p class="text-muted">Thanks <a href="https://twitter.com/kishoreyekkanti" target="_blank">Kishore
Yekkanti</a>, <a href="https://twitter.com/liuggio" target="_blank">Giulio De
Donato</a> for your review.</p>
<div class="post row">
<div class="col-md-12">
<div class="bs-callout bs-callout-info row">
<div class="row">
<div class="col-md-12">
<h2><a href="//gianarb.it/blog/docker-the-fundamentals" target="_blank">get "Docker the Fundamentals"</a> <small>by. Drive your boat as a Captain</small></h2>
</div>
</div>
<div class="row">
<div class="col-md-3">
<a href="//gianarb.it/blog/docker-the-fundamentals" target="_blank"><img src="/img/the-fundamentals.jpg" class="img-fluid" /></a>
</div>
<div class="col-md-9">
<p>
You can get the Chapter 2 of the book <a href="http://scaledocker.com" target="_blank">"Drive your boat as a Captain"</a> just leave click on the
cover and leave your email to receive a free copy.</p>
<p>This chapter is getting started with Docker Engine and the basic
concept around registry, pull, push and so on. It's a good way to start from
zero with Docker.</p>
</div>
</div>
</div>
</div>
</div>
Continuous Integration and silent checks. You are looking in the wrong place2016-11-18T10:08:27+00:00http://gianarb.it/blog/continuous-integration-and-silent-checks<p>Continuous Integration is a process of merging all developer working copies to
shared mainline several times a day. In practice is when you have in place a
system that allow you to trust all changes that all developers are doing in a
short period of time in order to have that code complaint and ready to be
pushed in production.</p>
<p>There are a lot of different way to do CI but I will stay focused on a very
important expect, you need a policy that contains a series of checks that you
can easy automate. All this steps persisted on every change allow you to mark
that new code as <code class="highlighter-rouge">ready</code>.</p>
<p>Automation is an important part to keep your integration continuous, usually what
people do is a human review of the code, if one or more people mark your code
as complaints and the continuous integration system is agree with them your code
can be merged. This is the unique manual step.</p>
<p>But let’s talk about what I call “Silent Checks” they are really one of the
best invention that I never saw. Silent Checks are like cigarettes, all know
that they are not so good but nobody cares.</p>
<p>Usually your CI system use exit code to understand if a check is good or bad,
your command come back with <code class="highlighter-rouge">0</code> in case of success or with another number if
something fails. Sometime you can find in your continuous integration checks
that put the status code in a silent mode. The check fails but it’s not important enough.</p>
<p><img class="img-fluid" src="/img/the-wolf-ci.jpeg" alt="continuous integration party" /></p>
<p>You have a check that runs but you are not asking people to care about the
result. Probably because it’s not important enough. There are few disadvantages
about this approach:</p>
<ul>
<li>That check is making your job slow.</li>
<li>If the job doesn’t fail no one care about that optional check and your check
will never fail.</li>
<li>When a job fails you just need to scroll and jump over all the logs generated
by the optional check. They produce a very long logs because usually they
fails. There is more, usually your coworkers forget about this check and they
ping you about that errors.</li>
</ul>
<p>Analyse your code is very important but there are other strategies that
you can use to avoid this inconvenient. Usually the silent checks are in place
in a period of migration, maybe they are important to monitor how it is going.
They are just in the bad position.
You can move them in a separated job, collected them and analyse what you need
to analyse and monitoring trends about how your team works.</p>
<p>I saw a TEDx Talk by Adam Tornhill. He talked about Analyzing Software with
forensic psychology. This topic is great! You can get a lot of informations
about your application from who is writing that code.</p>
<div style="text-align:center">
<iframe width="640" height="360" src="https://www.youtube.com/embed/qJ_hplxTYJw" frameborder="0" allowfullscreen=""></iframe>
</div>
<p>Trends and monitoring not just to understand how your application works but
they are fundamentals to understand how your team is working, how they
feel and also to catch how your codebase is moving. They are really important
and if you are strong enough to have a good monitoring system for that metric
you are really in a good position! You just need to understand that insert
them into the continuous integration flow is not a good idea.</p>
Docker Bench Security2016-11-15T10:08:27+00:00http://gianarb.it/blog/Docker-Security-Benchmark<p>Frequently, best practices help you to have a safe environment,
<a href="https://github.com/docker/docker-bench-security">docker-bench-security</a> is an
open source project that runs in a container and scans your environment to
report a set of common mistakes like:</p>
<ul>
<li>Your kernel is too old</li>
<li>Your docker is not up to date</li>
<li>Some Docker daemon configurations are not good enough to run a production environment</li>
<li>Your container runs 2 processes</li>
<li>and others</li>
</ul>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>It’s a great idea to run it at some stage in each host to have an idea about
the status of your environment. To do that you can just use this command when
running a container</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>docker run <span class="nt">-it</span> <span class="nt">--net</span> host <span class="nt">--pid</span> host <span class="nt">--cap-add</span> audit_control <span class="se">\</span>
<span class="nt">-v</span> /var/lib:/var/lib <span class="se">\</span>
<span class="nt">-v</span> /var/run/docker.sock:/var/run/docker.sock <span class="se">\</span>
<span class="nt">-v</span> /usr/lib/systemd:/usr/lib/systemd <span class="se">\</span>
<span class="nt">-v</span> /etc:/etc <span class="nt">--label</span> docker_bench_security <span class="se">\</span>
docker/docker-bench-security
</code></pre></div></div>
<p>A good way to start is your run it in your local environment. Run the command
and check what you can do to make your local environment safe. This tool is
open source on GitHub and it’s also a great example of collaboration and how a
community can share experiences to help other members to improve an
environment. This is a partial output:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Initializing Thu Nov 24 21:35:24 GMT 2016
<span class="o">[</span>INFO] 1 - Host Configuration
<span class="o">[</span>WARN] 1.1 - Create a separate partition <span class="k">for </span>containers
<span class="o">[</span>PASS] 1.2 - Use an updated Linux Kernel
<span class="o">[</span>PASS] 1.4 - Remove all non-essential services from the host - Network
<span class="o">[</span>PASS] 1.5 - Keep Docker up to date
<span class="o">[</span>INFO] <span class="k">*</span> Using 1.13.01 which is current as of 2016-10-26
<span class="o">[</span>INFO] <span class="k">*</span> Check with your operating system vendor <span class="k">for </span>support and security maintenance <span class="k">for </span>docker
<span class="o">[</span>INFO] 1.6 - Only allow trusted users to control Docker daemon
<span class="o">[</span>INFO] <span class="k">*</span> docker:x:999:gianarb
<span class="o">[</span>WARN] 1.7 - Failed to inspect: auditctl <span class="nb">command </span>not found.
<span class="o">[</span>WARN] 1.8 - Failed to inspect: auditctl <span class="nb">command </span>not found.
<span class="o">[</span>WARN] 1.9 - Failed to inspect: auditctl <span class="nb">command </span>not found.
<span class="o">[</span>INFO] 1.10 - Audit Docker files and directories - docker.service
<span class="o">[</span>INFO] <span class="k">*</span> File not found
<span class="o">[</span>INFO] 1.11 - Audit Docker files and directories - docker.socket
<span class="o">[</span>INFO] <span class="k">*</span> File not found
</code></pre></div></div>
<p>Sometime to have a good result you just need to run a single command.</p>
<p>This article is part of “Drive your boat like a Captain”. It’s a book about
Docker in production, how manage a cluster of Docker Engine with Swarm and what
it means to manage a production environment today.</p>
<p>Keep in touch to receive news about the book
<a href="http://scaledocker.com">scaledocker.com</a>. If you are looking for a Docker
Getting Started you can also look on the first chapter that I released <a href="http://localhost:4000/blog/docker-the-fundamentals">Docker
The
Fundamentals</a></p>
Chef Server startup notes2016-11-10T10:08:27+00:00http://gianarb.it/blog/chef-server-startup-notes<p>I worked with different provisioning tools and configuration managers in the
last couple of years: Chef, Saltstack, Puppet, Shell, Python, Terraform.
Everything that was allowing me to make automation and describe my
infrastructure as a code.</p>
<p>I really think that this is the correct street and every companies need to stop
to persist random commands in a server:</p>
<ul>
<li>The code used to describe your infrastructure is reausable.</li>
<li>The code is a good backup and you can put it in your repository to study of
it changed and manage rollbacks.</li>
<li>Your servers become collaborative and your team can review what you do.</li>
</ul>
<p>Chef is my first configuration manager, I started to use it with Vagrant few
years ago but I never had change to deep drive into it and into the full
chef-server configuration from scratch.</p>
<p>I had this change few days ago and I am here to share some notes. I used
digitalocean to start 1 Chef Server and two nodes, during this post I am not
focused about the recipe and cookbook syntax but I will share some commands and
notes that I took during my test to start and configure a Chef Server.</p>
<p>First of all <code class="highlighter-rouge">doctl</code> is the command line application provided by digitalocean
to manage droplets and everything, I used that tool to start my droplets.</p>
<p>The Chef Server doesn’t run in little box, we need 2gb RAM, I tried with small
size but nothing was working, the installation process gone out of memory very
soon. Thanks Ruby.</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>doctl compute droplet create chef-server <span class="se">\</span>
<span class="nt">--region</span> ams2 <span class="nt">--size</span> 2gb <span class="nt">--image</span> 20385558 <span class="se">\</span>
<span class="nt">--access-token</span> <span class="nv">$DO</span> <span class="nt">--ssh-keys</span> <span class="nv">$DO_SSH</span>
<span class="nv">$ </span>doctl compute droplet create n1 <span class="se">\</span>
<span class="nt">--region</span> ams2 <span class="nt">--size</span> 512mb <span class="nt">--image</span> 20385558 <span class="se">\</span>
<span class="nt">--access-token</span> <span class="nv">$DO</span> <span class="nt">--ssh-keys</span> <span class="nv">$DO_SSH</span>
</code></pre></div></div>
<p><code class="highlighter-rouge">$DO</code> contains my digitalocean access key and <code class="highlighter-rouge">$DO_SSH</code> the id of the ssh key
to log into the servers. You can leave the last one empty and you will receive
an email with the password.</p>
<p>When the process is gone you will be able to copy the ip of the chef-server and go into it.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>doctl compute droplet <span class="nb">ls
</span>ID Name Public IPv4 Public IPv6 Memory VCPUs Disk Region Image Status Tags
30cw4230 chef-server 2gb 1 20 ams2 Debian 8.6 x64 new
q0514230 n1 512 1 20 ams2 Debian 8.6 x64 new
</code></pre></div></div>
<p>This provisioning script installs Chef-Server from the official deb package and
also install chef-manage. Chef-manage provides a nice web interface to manage
users, cookbooks and everything is stored into the server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /tmp
<span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get install <span class="nt">-y</span> unzip curl
curl <span class="nt">-LS</span> https://packages.chef.io/stable/ubuntu/16.04/chef-server-core_12.9.1-1_amd64.deb <span class="nt">-o</span> chef-server-core_12.9.1-1_amd64.deb
<span class="nb">sudo </span>dpkg <span class="nt">-i</span> chef-server-core_12.9.1-1_amd64.deb
<span class="nb">sudo </span>chef-server-ctl reconfigure
</code></pre></div></div>
<p>Our server is up an running reachable over HTTPS (port 443). This configuration
is just for testing purpose. It’s not a good practice leave a Chef Server public
as we are doing. It’s a better idea to close it under a VPN for example.</p>
<p>Chef supports an authentication and authorization level based on users and
companies. We are creating a new user called <code class="highlighter-rouge">Gianluca Arbezzano</code> with username
<code class="highlighter-rouge">ga@thumpflow.com</code> and as password <code class="highlighter-rouge">hellociaobye</code>.
We are also create an organization and we are associating user with org.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chef-server-ctl user-create gianarb Gianluca Arbezzano ga@thumpflow.com <span class="s1">'hellociaobye'</span> <span class="nt">--filename</span> /root/gianarb_test.pem
chef-server-ctl org-create tf <span class="s1">'ThumpFlow'</span> <span class="nt">--association_user</span> gianarb <span class="nt">--filename</span> /root/tf-validator.pem
chef-server-ctl org-user-add tf gianarb
</code></pre></div></div>
<p>At this point we can configure a nice UI for our Chef Server with this simple
commands:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>chef-server-ctl install chef-manage
<span class="nb">sudo </span>chef-server-ctl reconfigure
<span class="nb">sudo </span>chef-manage-ctl reconfigure <span class="nt">--accept-license</span>
</code></pre></div></div>
<p>Chef-Server works with the concept of Organization and User. The organization
is a group of users that share cookbooks, rules and so on. Users can update
cookbooks and there is also a set of permission to manage access on particular
resources like:</p>
<ul>
<li>Add a new node</li>
<li>Syncronize cookbook with the server</li>
<li>add new users</li>
</ul>
<p>At this point we have one user with its own key and credential. You can come
back into the UI and use username (gianarb) and password (hellociaobye) to
login in. The key (–filename) is used to configure knife and encrypt
communication between client and server. There are 3 main actors and this
point that we need to know:</p>
<ul>
<li>Chef-Server contains all our recipes, cookbooks and it’s the brain of the cluster.</li>
<li>Nodes are all servers configurated by Chef.</li>
<li>Workstation are usually enable to syncronize, update cookbooks. For example
Jenkins or your Continuous Integration System after every new commit can push
every changes into the server.</li>
</ul>
<p>Chef Server has a HTTP api and <code class="highlighter-rouge">knife</code> is a CLI that provide an easy
integration for your node and workstation. With this command we are installing
knife. You can do it in your local environment, to become a workstation and into
the server. (it’s usually a god practice create a user, we are doing everything
as root right know but it’s BAD! don’t be bad!).</p>
<p>We have two certificate one is <code class="highlighter-rouge">gianarb_test.pem</code> and it’s identify a specific
user, we need to generate our for every workstation/member of the team and the
<code class="highlighter-rouge">validation_client</code> represent the organization, it could be the same across
multiple users.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-O</span> <span class="nt">-L</span> http://www.opscode.com/chef/install.sh
bash ./install.sh
</code></pre></div></div>
<p>You can copy paste the 2 keys into the local machine and run this command that
will drive your into the process to create a <code class="highlighter-rouge">~/.chef/knife.rb</code> file that your
cli uses to communicate with the chef server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>knife configure
</code></pre></div></div>
<p>This is an example of generated knife configuration file that I did in my
server. I lose times to understand the <code class="highlighter-rouge">chef_server_url</code> it contains the
hostname of the server but also the `/organization/<organization_short_name>'
be careful about this or knife will come back with an HTML response in your terminal.</organization_short_name></p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">log_level</span> <span class="ss">:info</span>
<span class="n">log_location</span> <span class="no">STDOUT</span>
<span class="n">node_name</span> <span class="s1">'gianarb'</span>
<span class="n">client_key</span> <span class="s1">'/root/gianarb_test.pem'</span>
<span class="n">validation_client_name</span> <span class="s1">'tf-validator'</span>
<span class="n">validation_key</span> <span class="s1">'/root/tf-validator.pem'</span>
<span class="n">chef_server_url</span> <span class="s1">'https://chef-server:443/organizations/tf'</span>
<span class="n">syntax_check_cache_path</span> <span class="s1">'/root/.chef/syntax_check_cache'</span>
<span class="n">cookbook_path</span> <span class="p">[</span><span class="s2">"/home/gianarb/git/chef-pluto/cookbooks"</span><span class="p">]</span>
<span class="n">ssl_verify_mode</span> <span class="ss">:verify_none</span>
</code></pre></div></div>
<p>The last 2 commands download and validate the SSH certificate because in the
default configuration the CA is unofficial and we need to force our client to
trust the cert.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>knife ssl fetch
knife ssl check
</code></pre></div></div>
<p>Know that we did that in our server and also in our local environment we can
clone <a href="https://github.com/gianarb/chef-pluto">chef-pluto</a> a repository that contains recipes, rules and cookbooks to
configure our node, we need to syncronize it into the server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git clone git@github.com:gianarb/chef-pluto.git /home/gianarb/git/chef-pluto/chef-pluto
<span class="nb">cd</span> /home/gianarb/git/chef-pluto/chef-pluto
knife update /
</code></pre></div></div>
<p>The last command update all our repository into the chef server. You can log in
into the web ui and see the <code class="highlighter-rouge">micro</code> cookbook and the <code class="highlighter-rouge">power</code> rule.</p>
<p><a href="https://github.com/gianarb/micro">micro</a> is an application that I wrote in go and it just expose the ip of the
machine. It’s a binary and the cookbook downloads and starts it, pretty
straightforward.</p>
<p>At this point we need to make a provisioning of our first node, usually is the
server that install and start the Chef Client into the node, what we can do
it’s store a private key into the server to allow chef to connect to the node.
I copied the digitalocean private key into the server (~/do), from security
point of view you can create a dedicate one. You can also use the -P option if
you are not using an ssh-key to run this example.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>knife bootstrap &lt;ip-node&gt; <span class="nt">-N</span> node1 <span class="nt">--ssh-user</span> root <span class="nt">-r</span> <span class="s1">'role[power]'</span> <span class="nt">-i</span> ~/do
</code></pre></div></div>
<p>If everything it’s good you can reach the application from port <code class="highlighter-rouge">8000</code> into the
browser. The log is something like:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>knife bootstrap 95.85.52.211 <span class="nt">-N</span> testNode <span class="nt">--ssh-user</span> root <span class="nt">-r</span> <span class="s1">'role[power]'</span> <span class="nt">-i</span> ~/do
Doing old-style registration with the validation key at /root/tf-validator.pem...
Delete your validation key <span class="k">in </span>order to use your user credentials instead
Connecting to 95.85.52.211
95.85.52.211 <span class="nt">-----</span><span class="o">&gt;</span> Existing Chef installation detected
95.85.52.211 Starting the first Chef Client run...
95.85.52.211 Starting Chef Client, version 12.15.19
95.85.52.211 resolving cookbooks <span class="k">for </span>run list: <span class="o">[</span><span class="s2">"micro"</span><span class="o">]</span>
95.85.52.211 Synchronizing Cookbooks:
95.85.52.211 - micro <span class="o">(</span>0.1.0<span class="o">)</span>
95.85.52.211 Installing Cookbook Gems:
95.85.52.211 Compiling Cookbooks...
95.85.52.211 Converging 2 resources
95.85.52.211 Recipe: micro::default
95.85.52.211 <span class="k">*</span> remote_file[Download micro] action create_if_missing <span class="o">(</span>up to date<span class="o">)</span>
95.85.52.211 <span class="k">*</span> service[Start micro] action start
95.85.52.211 - start service service[Start micro]
95.85.52.211
95.85.52.211 Running handlers:
95.85.52.211 Running handlers <span class="nb">complete
</span>95.85.52.211 Chef Client finished, 1/2 resources updated <span class="k">in </span>02 seconds
</code></pre></div></div>
<p>knife started the client, syncronized cookbooks, it assigned the <code class="highlighter-rouge">power</code> role
at the node and run the correct recipes. Your server is ready and you can
create and delete nodes to make your infrastructure complex how much you like.</p>
<p>Chef is quite old and it’s in ruby (the first one could be a plus but the
second one no really) but it continue to be a good way to make a provisioning o
your infrastructure. Lots of people moved to Ansible but the agent that they
reject offer a very good orchestration feature that it’s something that I
usually search.</p>
<p>I worked with StalStack and it’s very nice, the syntax is easy
and it seems less expensive in terms of configuration, resources and setup but
I am not really sure about the YAML specification. I am not a ruby developer
and I don’t love the ruby syntax but in the end is a programming languages and
I am doing infrastructure as a code.</p>
Docker The fundamentals2016-08-25T12:08:27+00:00http://gianarb.it/blog/docker-the-fundamentals<p>I am writing a book about Docker SwarmKit and how manage a production
environment for your containers.</p>
<p>The second chapter of the book is a Getting Started about Docker, it covers
basic concepts about what container means and it’s a started point to
understand the concepts expressed into the book.</p>
<h2>Drive your boat like a Captain.
<small>Docker in production</small></h2>
<p>The book is work in progress but you can find more information into the site
<a href="http://scaledocker.com">scaledocker.com</a>.</p>
<p>To receive the first chapter free leave your email and if you like your twitter account:</p>
<div class="row">
<div class="col-md-6">
<img src="/img/the-fundamentals.jpg" class="img-fluid" />
</div>
<div class="col-md-4">
<form id="get-chapter">
<div class="form-group">
<label for="exampleInputEmail1">Email address *</label>
<input type="email" class="form-control" required="required" id="email" placeholder="Email" />
</div>
<div class="form-group">
<label for="exampleInputPassword1">Twitter</label>
<input type="title" class="form-control" id="twitter" placeholder="@gianarb" pattern="^@.*" />
<p class="help-block">The first letter needs to be a @</p>
</div>
<p class="text-success get-chapter-thanks">Check your email! Thanks!</p>
<p class="text-warning get-chapter-sorry"><span class="err-text"></span>.
Please notify the error with a comment or with an email</p>
<button class="btn btn-default">Get your free copy</button>
</form>
</div>
</div>
<h2>Contents</h2>
<ol>
<li>Introduction</li>
<li>Install Docker on Ubuntu 16.04</li>
<li>Install Docker on Mac</li>
<li>Install Docker on Windows</li>
<li>Run your first HTTP application</li>
<li>Docker engine architect</li>
<li>Image and Registry</li>
<li>Docker Command Line Tool</li>
<li>Volumes and File Systems 20</li>
<li>Network and Links</li>
<li>Conclusion</li>
</ol>
<p>Enjoy your reading and leave me a feedback about the chapter!</p>
<script>
(function() {
$(".get-chapter-thanks").hide();
$(".get-chapter-sorry").hide();
var api = "https://1lkdtyxdx4.execute-api.eu-west-1.amazonaws.com/prod";
$("#get-chapter button").click(function(eve) {
eve.preventDefault()
$(".get-chapter-thanks").hide();
$(".get-chapter-sorry").hide();
var requestChapter = $.ajax({
"url": api+"/the-fundamentals",
"type": 'post',
"data": {
email: $("#email").val(),
twitter: $("#twitter").val()
},
"dataType": 'json',
"contentType": "application/json"
});
requestChapter.done(function() {
$(".get-chapter-thanks").show();
});
requestChapter.fail(function(data) {
$('.err-text').html("["+data.responseJSON.code+"]"+ data.responseJSON.text);
$(".get-chapter-sorry").show();
});
});
})();
</script>
Be smart like your healthcheck2016-08-25T12:08:27+00:00http://gianarb.it/blog/be-start-like-your-healthcheck<p>I am not a doctor, I am a Software Engineer and this is a tech post! You can
continue to read!</p>
<p>To monitor monolithic what we usually do is install a tool
like <a href="https://www.nagios.org/">Nagios</a> to centralize all our metrics and to
stay in touch with our infrastructure and our application. In a distributed
system with more that one services with own metrics the situation is totally
different. This about how it’s more dynamic respect a monolithic. Containers
or VM that scale up and down and that move around the network, Nagios is a good
solution to check if our new service after a deploy is safe and ready to be
attached into the production pool? I love a talk made by <a href="https://github.com/kelseyhightower">Kelsey
Hightower</a> during the Monitorama event, he
speak about healthcheck watch him to follow a <a href="https://vimeo.com/173610242">great demo</a>!</p>
<p>Healthcheck is an API that your service exposes to share it’s status, if you
make it really start it’s a good tool to understand the situation of your
service with just a call. A service could be ready or not and it’s in the best
situation to communicate its status. It’s a like a patient, you need to ask
him all what you need to make the best diagnosis and take a decision about it.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>We can stay focused on a REST service, it exposes an API under the route
/health. The response could has two different Status Code:</p>
<ul>
<li>200 if all it’s good and you service is ready</li>
<li>500 it there is something wrong and your service is not ready</li>
</ul>
<p>To make an smart HealthCheck what do we need to check?</p>
<p>This is a real implementation:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?php</span>
<span class="k">echo</span> <span class="mi">1</span><span class="p">;</span>
</code></pre></div></div>
<p>It’s better that nothing but we are looking for something smart! We need to
check all dependendencies that our service has and it’s for this reason that
the service itself is the best actor because it knows what it need to be ready.
I wrote a demo service, the name is <a href="https://github.com/gianarb/micro/blob/master/handle/health.go">micro</a>, it’s in go and
the version 2 use
mysql.</p>
<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">func</span><span class="x"> </span><span class="n">Health</span><span class="p">(</span><span class="n">username</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="n">password</span><span class="x"> </span><span class="kt">string</span><span class="p">,</span><span class="x"> </span><span class="n">addr</span><span class="x"> </span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="k">func</span><span class="p">(</span><span class="n">w</span><span class="x"> </span><span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="x"> </span><span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">res</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">healtResponse</span><span class="p">{</span><span class="n">Status</span><span class="o">:</span><span class="x"> </span><span class="no">true</span><span class="p">}</span><span class="x">
</span><span class="n">httpStatus</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="m">200</span><span class="x">
</span><span class="n">dsn</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">fmt</span><span class="o">.</span><span class="n">Sprintf</span><span class="p">(</span><span class="s">"%s:%s@tcp(%s:3306)/micro"</span><span class="p">,</span><span class="x"> </span><span class="n">username</span><span class="p">,</span><span class="x"> </span><span class="n">password</span><span class="p">,</span><span class="x"> </span><span class="n">addr</span><span class="p">)</span><span class="x">
</span><span class="n">ddb</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">sql</span><span class="o">.</span><span class="n">Open</span><span class="p">(</span><span class="s">"mysql"</span><span class="p">,</span><span class="x"> </span><span class="n">dsn</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">ddb</span><span class="o">.</span><span class="n">Ping</span><span class="p">();</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">res</span><span class="o">.</span><span class="n">Status</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="no">false</span><span class="x">
</span><span class="n">res</span><span class="o">.</span><span class="n">Info</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="kt">string</span><span class="p">{</span><span class="s">"database"</span><span class="o">:</span><span class="x"> </span><span class="n">err</span><span class="o">.</span><span class="n">Error</span><span class="p">()}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">c</span><span class="p">,</span><span class="x"> </span><span class="n">_</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">json</span><span class="o">.</span><span class="n">Marshal</span><span class="p">(</span><span class="n">res</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">res</span><span class="o">.</span><span class="n">Status</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="no">false</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">httpStatus</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="m">500</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="s">"%s called /health"</span><span class="p">,</span><span class="x"> </span><span class="n">r</span><span class="o">.</span><span class="n">Host</span><span class="p">)</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">WriteHeader</span><span class="p">(</span><span class="n">httpStatus</span><span class="p">)</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">Header</span><span class="p">()</span><span class="o">.</span><span class="n">Set</span><span class="p">(</span><span class="s">"Content-Type"</span><span class="p">,</span><span class="x"> </span><span class="s">"application/json"</span><span class="p">)</span><span class="x">
</span><span class="n">w</span><span class="o">.</span><span class="n">Write</span><span class="p">(</span><span class="n">c</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre></div></div>
<p>Doesn’t matter how many dependencies you service has, you need to check all of
them, databases, other services that it uses. In my case I decided to add a
key-value field, I called it <code class="highlighter-rouge">info</code>, it contains some information about whether
mysql is or is not not working, in order to make the debug flow easy. If the
service that you are checking has an healthcheck you are lucky! You can use
that entrypoint to know if your dependency is fine. If you are not so lucky if
you can create a wrapper or just check if you can reach the service, in my case
I just tried to connect to mysql in order to know if my network supports me! I
also using the correct database name in order to avoid edge case like “mysql is
on but the database doesn’t exist”.</p>
<p>The ecosystem supports healthchecks! Nginx looks it to know if a server is
reachable, if the health check doesn’t work for a while it just make the server
out for few times. Same for Kubernetes, Swarm and Docker. Docker provides a
library in go an <a href="https://github.com/docker/go-healthcheck">healthcheck
framework</a> that you can use in your
applications, it is also used in Docker 1.12.</p>
<p>You can describe in your Dockerfile an HealthCheck</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HEALTHCHECK CMD ./cli health
</code></pre></div></div>
<p>If the exit code is 0 Docker marks you container like healthy if it’s different like unhealthy.
Very easy and flexible, you can check your REST healthcheck in this way</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>HEALTHCHECK --interval=30s --timeout=30s --retries=3 \
CMD curl -si localhost:8000/health | grep 'HTTP/1.1 200 OK' &gt; /dev/null
</code></pre></div></div>
<p><code class="highlighter-rouge">--interval</code> is the timing between two healthcheck, <code class="highlighter-rouge">--timeout</code> is used to mark
like unhealthy a service that doesn’t come back after 30s in this case,
<code class="highlighter-rouge">--retries</code> is the attempts to do before make a container unhealthy.</p>
<p>HealtCheck doesn’t replace traditional monitoring system but with a lot of
instances and services has a single point to check and understand the situation
after a deploy make your like easy and your products stable.</p>
Build opinions to become wise2016-08-25T12:08:27+00:00http://gianarb.it/blog/build-opinions-to-become-wise<p>I am Gianluca, I am 24 years old and I work as Software Engineer.</p>
<p>I like my work because there are really a lot of different kinds of Software
Engineer, mainly because you can work in pretty much every environment like
technology, food, cook, sport, fashion.</p>
<p>Also because there are a lot of stuff to do, if you like to work on a product,
you build features to make happy other people to buy a car or to read a news on
a online news paper.</p>
<p>If you like play with cables, racks, switchs you can work in a big or small
farm and design fancy datanceters.</p>
<p>But I like this work because a lot of people have opinions. There are opinions
built on top of big study or after a long experience but everyone is happy to
share them and use them to build a new product.</p>
<p>I am a good PHP developer, because I worked for a while with this language and
I tried to catch and verify good opinions from a lot of developers that come
from different parts of the world and from different experiences.</p>
<p>I am happy so share someone of them with you:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">Opinion\One</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">Good</span> <span class="p">{</span>
<span class="k">private</span> <span class="nv">$important</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">__contruct</span><span class="p">(</span><span class="nv">$somethingThatIReallyNeed</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-&gt;</span><span class="na">important</span> <span class="o">=</span> <span class="nv">$somethingThatIReallyNeed</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Usually I force myself (and when I can other people :P ) to inject objects from
contructor if our object (Good) can not work without them.</p>
<p>This is a good way to be sure that your object will be completed, because you can not forgot nothing!</p>
<p>I use Zend Framework from a lot of time and I remember one class
<code class="highlighter-rouge">ServiceLocatorAwareInterface</code>.</p>
<p>I have an opion, I hate this class.</p>
<p>If you implement this interface your service has a service locator! It’s
powerfull today that you need to finish your ticket and go away with a new
feature.
After months a lot of people improve this service and they start to use a lot
of other services without think about for what reason you built your service.
Just because it’s really simple get random services from service locator.</p>
<p>Be wise, don’t allow people to write bad code and use your constructor to inject your dependencies!!</p>
<p>I have also architecture opinions like:</p>
<p>When you think about “How can I resolve this problem?” you must start from
design pattern! They are documented and tested from a lot of developers and in
more use cases.</p>
<p>Imagine your colleague that come to you around 5pm to ask how an entire
libraries is designed, you can just reply “It’s just a SOAL client, see you
tomorrow!” and you go to play basketball.</p>
<p>Or if you are building an API, OAuth2 could be a good choice for your
anthentication service. It’s tested, there are a lot of clients and
documentations.
Your clients will be good to know that you are just using
Oauth2 and nothing strange.</p>
<p>Well, I am not here to share all my opinion in a single post, all of them are
just examples of what I mean for opinion.</p>
<p>An opinion is really important! But it’s just an opinion!</p>
<p>As a Software Engineer you need to have an opinion because every day people
will try to have opinion for you: Microservices, containers, one big
repository, golang/rust.</p>
<p>To build an opinion you need to make experience and to study it’s a big effort
but to be really wise you need also to stay ready to change your
opinion.</p>
<p>In my opinion this is the main difference between smart and wise people. I
prefer the second one!</p>
<div class="alert alert-success" role="alert">
Thanks for your review <a href="https://twitter.com/fntlnz" target="_blank">Lorenzo</a>!</div>
Watch demo about Docker 1.12 made during Docker Meetup2016-08-24T12:08:27+00:00http://gianarb.it/blog/docker-1-12--meetup-dublin<p>In August during the Docker Meetup I presented with a demo some new
features provided by Docker 1.12.</p>
<p>It’s an important release because it improves your experience with Docker
in production with an orchestration framework included into the product.</p>
<p>Docker provides a new set of commands to create a cluster of Docker
deamon and manage a production enviroment.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>It’s something like Kubernetes, Mesos, Swarm but it is included and
built-in Docker.</p>
<p>I wrote an article about it few months ago <a href="http://gianarb.it/blog/docker-1-12-orchestration-built-in">“Docker 1.12 orchestration
built-in”</a>.</p>
<p>In this demo I do an introduction of some new features like:</p>
<div style=" text-align: center;">
<iframe width="420" height="315" src="https://www.youtube.com/embed/h7a7vhzjElo" frameborder="0" allowfullscreen=""></iframe>
</div>
<ul>
<li>How create a SwarmMode docker cluster</li>
<li>What is a service? What tasks means?</li>
<li>How Docker SwarmKit manage a node down?</li>
<li>I tried to show the HealthCheck feature :)</li>
<li>How docker swarmkit manage containers update</li>
<li>service discovery</li>
</ul>
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=90cae01e86" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">News from Docker Planet</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_90cae01e86" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
<div class="row">
<div class="col-md-12">
<a href="//gianarb.it/planet/docker.html"><b>Docker Planet</b></a> is a collection of posts, events and other
information about Docker and all the container ecosystem. Read other
articles and stay in touch, subsbrite to the newsletter.
</div>
</div>
“Microservices and common parts"2016-08-14T12:08:27+00:00http://gianarb.it/blog/services-and-common-parts<p>Changing my glossary and replacing the concept of application with
service could be a buzzword but this takes me to built a
new approach to my work.</p>
<p>Nowadays many products require more services than before to
work: perhaps they could be modules, libraries directly integrate in one or
more services or applications that communicate and provide a feature, it
doesn’t matter cause in any case the product’ll have some dependences.</p>
<p>If you start to follow this path, a lot of redundant concepts will show up in your
product:</p>
<ul>
<li>Monitoring</li>
<li>Logging</li>
<li>Authentication</li>
<li>Scaling</li>
<li>High availability</li>
<li>Distribution</li>
<li>Testing (unit, functional, integration..)</li>
<li>And others</li>
</ul>
<p>Some of them, like monitoring or logs, require architecture and tools
selection: you can use some B2B tools or host something in house. It’s not
only a problem of tooling, the other face of this “redundant money” is how your
services can communicate logs, metrics with outside in a clean and reusable
way.</p>
<p>In this post I will try to share the common part of a
microservices ecosystem and some possible approach to solve this issues.</p>
<h2 id="logging">Logging</h2>
<p>All applications require a good and strong log system. There are few
libraries able to help you in managing this section but the minimum requirement, in my opinion, includes:</p>
<ul>
<li>Support for multiple stream: usually, I use stdout or file and I move
them in a database with a separate pipeline, but a lot of good libraries allow you to manage your logs in different collectors.</li>
<li>Different layers like: INFO, DEBUG, WARNING, FATAL.</li>
<li>Provide a way to change this layer runtime, for example with a RPC call.</li>
</ul>
<p>The third point is really important: if your application start to have a big
traffic, the amount of logs you must manage will be relevant; so, changing this
level runtime allows you to manage the amount of logs that you store and, for example, allows DEBUG
information only if you need to do some specific debugging in production. This strategy save storage and money.</p>
<p>There are a lot of services and open source tools able to manage and storage this data. The real issue is decide which street follow.</p>
<p>Are you interested to manage your logs or it’s a big effort for your company? You can move all to log entries and forgot about elastic search and kibana
and similar in this case. Think about your environment and catch the best solution. Remember that it could be just a temporary solution. When you start a business you have different thoughts, start slim and easy.</p>
<h2 id="monitoring">Monitoring</h2>
<p>Several services require several time and energy to be monitored and to be
maintained alive.</p>
<p>The best way to do that is with a time series
database like prometheus, InfluxDB or other as a service solution like NewRelic
or AppDynamics.</p>
<p>The real problem is how your application can provide
metrics readable and usable from external systems. You can find a very good solution to this problem in Docker: they provide different streams and events to grab this kind of informations.</p>
<p>If you take a lot on how it manage this part
you can implement a good system in your application. A stream of events is
also a good API to allow other services to enjoy features provided from your
service.</p>
<h2 id="heahtcheck">Heahtcheck</h2>
<p>Understand with a single request if your application has all what it needs to
work is really important.</p>
<p>The microservices ecosystem contains a lot of micro applications that change and have dependencies to work. How can you understand if all your system is up and runs without spend a lot of time?</p>
<p>You can create for each service the call <code class="highlighter-rouge">/heath</code> that return 200 if all it’s fine and 500 if there is something that it’s not working properly.</p>
<p>During a release you can use this endpoint in order to understand if your
service is ready to be attached into the production pool.</p>
<p>In practice, if you have one service called Users that depends from MySQL and
from another service like Emailer, the health entrypoint for Users’s service
will check whether it can connect to the MySql and also you can call <code class="highlighter-rouge">/health</code> for
Emailer in order to check if the service is up.</p>
<p>Your orchestration and deploy framework can check after each deploy if the health is up and running and manage your release, it can revert or it doesn’t include your new release into the production pool.</p>
<h2 id="authentication">Authentication</h2>
<p>Your microservice is not public, sometime you have a set of firewall’s rules or
a strong network settings to manage the security of your environment but for other
services the authentication layer is a requirement and usually there are few
services that need to know which is the identity of the user that is persisting
an action.</p>
<p>Think about a To Do service, it need to know the identity of the user in order
to fetch the correct items.</p>
<p>For this reason this layer could be common between your services and it’s also a
critical section of your architecture because usually from it depends the security of
your application and users.</p>
<p>Oauth2 is a framework to manage authentication, I recommend it
because it has a documentation already done, it’s a standard.
You don’t reivent anything, there are a lot of libraries and use cases about it that make it solid, flexible and reusable.</p>
<h2 id="automation-and-deploy">Automation and Deploy</h2>
<p>A good layer of automation is important in every ecosystem to make your work less
bored but also to decrease chance for a human to make a mistake during a
repetitive task.</p>
<p>If you are thinking about a microservices ecosystem all this problems are
multiplied for a big number of applications.</p>
<p>Without a good layer of automation and a good deploy’s flow you will spend all
your day to put line of code in production without have time to stay focused on
new features or other business’s requests.</p>
<h2 id="documentation">Documentation</h2>
<ul>
<li>Describe the topology of your ecosystem,</li>
<li>how match microservices you have?</li>
<li>where they are and how they are distributed across your datacenters</li>
<li>Make it extensible and easy to read and update.</li>
<li>How a single service works?</li>
<li>Which APIs it expose</li>
<li>how another service can communicate with it.</li>
<li>Single dependencies for each microservices is also important to know.</li>
</ul>
<p>All common part like, logs, auth, metrics help you to have a
common documentation easy to maintain, read and implement but for each service
you must provide a specific documentation because all it’s clear today but
between few months when you worked on ten other services the situation could be
really different.</p>
<p>One of the goal about microservices is the possibility to add
and integrate them easily. Documentation is one of the goal to make this
possible and efficient.</p>
<h2 id="communication-layer">Communication Layer</h2>
<p>A lot of companies have one communication layer in the
environment, JSON and REST. It’s a good choice, easy to implement and there are
also a lot of tools to test, document and create client libraries.</p>
<p>But HTTP/REST is not the unique way to expose features out of your service, this is
really important to know.</p>
<p>There are other efficient and less expensive solution, binary protocol is
one of them.</p>
<p>For all this topics we can stay here to speech for years for this reason I have in
plan other posts to analyze some points better.</p>
<p>Please let me know if in your experience there are other common part between
your services.</p>
<div class="row">
<div class="col-md-3">
<img src="/img/distributed_system_planet.png" class="img-fluid" />
</div>
<div class="col-md-9">
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=c4f95a57f7" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">Stay in touch to read about Distributed System</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_c4f95a57f7" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
</div>
</div>
<!--End mc_embed_signup-->
What Distributed System means2016-07-12T16:08:27+00:00http://gianarb.it/blog/distributed-system-means<p>I choose to put my experience about distributed system in a serie of blog
posts on which I’ll cover different topics.</p>
<p>I will speak about service discovery, micro services, container, virtual
machines, schedulers, cloud, scalability and latency I hope to have, at the end
of this experience a good number of posts in order to share what I know and how
I work and approach this kind of challenges.</p>
<p>In first I will not speak about nothing new, in fact distributed system means:</p>
<blockquote>A distributed system consists of a collection of autonomous computers,
connected through a network and distribution middleware, which enables
computers to coordinate their activities and to share the resources of the
system, so that users perceive the system as a single, integrated computing
facility.
<p><a href="http://www0.cs.ucl.ac.uk/staff/ucacwxe/lectures/ds98-99/dsee3.pdf" target="_blank">Wolfgang Emmerich, 1997</a></p>
</blockquote>
<p>Internet is a distributed system, you infrastructure is usually a distributed
system if you follow the minimum requirements to make high availability for
your services.</p>
<p>In first of all I love what service means, your application is a service,
microservices is just a way to remind to people that a little application is
easy to maintain, deploy and control but the idea in my opinion is just make
something autonomous and useful for your customers. Sometimes your customer is
a human in other case could be another service provided by yourself or from a
third-party, it is not really important. It’s important that your service must
be ready to communicate with the extern.</p>
<p>Distributed your system is important to make it available, if you close your
service inside a single datacenter in a single part of the world you take the
risk to make it unavailable in case of problem in that particular area, if you
distribute your service in different location you are increasing the chances to
stay up.</p>
<p>You are also mitigating the latency around your system because you are bringing
your application near your customers and if you have a world wide traffic
that’s param is really important.</p>
<p><img alt="Internet Global Submarine map" src="/img/global-submarine-cable.jpg" class="img-fluid" /></p>
<p>This is the map of the submarine cable (2014) and all know that internet is not
in the air and serve different point in the world require different amount of
time to have a response and also is not just a problem of distance but traffic
and quality of the network have them weight. Akamai is a expert about this
topic, he provide a service of content delivery (CDN) and also it’s a
monitoring system for the status of the network, they provide different data,
one of them describe the <a href="https://www.akamai.com/us/en/solutions/intelligent-platform/visualizing-akamai/real-time-web-monitor.jsp">high level status of Internet</a>.</p>
<p>Virtualisation, container, cloud computing and in general the low price to
design an infrastructure and the growth of internet’s users allow little
company with a little budget to create something of stable, secure and
available in different part of worlds. I think that for this reason micro
services and distributed system start to have a big impact in the industry.</p>
<p>A good exercise to understand the current situation could be design a little
infrastructure cross provider in multi datacenter to support a normal blog,
with a database and an application. With a couple of servers on different cloud
providers you can create an high available and distributed system across
multiple datacenter and avoid a lot of point of failure like: Geography
disaster Provider errror…</p>
<p>Docker, openstack, AWS, Consul, Prometheus, Elasticsearch, MongoDB are just a
set of products that help us to create something really stable and useful.
Continue Delivery, High Availability, disaster recovery, monitoring, Continuous
Integration, reliable are a subset of topic that you must resolve when you
think about distributed system because you can not care about where the
instances of your applications are around the world and the network is not a
paradise of stability. Microservices helps you to create better and stable
application, allow your company to create more rooms for more developers and to
replace single pieces and features but they create other kind of problems like
architecture complexity, good knowledge of in different layers (DevOps point of
view), network and chain of failures. All the topics that we already know must
be adapter for this new architecture, monitoring, logging, deploy.</p>
<div class="row">
<div class="col-md-3">
<img src="/img/distributed_system_planet.png" class="img-fluid" />
</div>
<div class="col-md-9">
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=c4f95a57f7" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">Stay in touch to read about Distributed System</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_c4f95a57f7" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
</div>
</div>
<!--End mc_embed_signup-->
Symfony and InfluxDB to monitor PHP application2016-07-02T10:08:27+00:00http://gianarb.it/blog/symfony-and-influxdb-to-monitor-php-applications<p>Symfony is one of the most famous PHP Frameworks in use right now, today we are
going to use it to understand how much is important to know how one our
features performs. We don’t monitor CPU usage, I/O disk or the number of
server errors but we are monitoring the final feature from the business point
of view.
This approach is very important because understanding which is the
impact of a new release on a specific, critical feature is the best way to
understand how a customer use our service.</p>
<p>In this article we are implementing
a monitor for one of the most diffuse business requirements, the
authentication.</p>
<p>In order to understand how many people try to do a login, and
track how many of them perform a wrong authentication and use this metrics to
understand how the system evolves.
Sometimes happens that right after a deploy
the number of wrong logins grows faster than usual, this could be a sign that
the feature doesn’t work as expected. We begin from the standard Symfony
application</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>composer create-project symfony/framework-standard-edition influxdb_app
<span class="nv">$ </span><span class="nb">cd </span>influxdb_app/
<span class="nv">$ </span>php bin/console server:run</code></pre></figure>
<p>We create a route under authorization (/admin), it is private and only admin
users are allowed to see that and one public homepage (/).</p>
<p>You can follow the official tutorial, or this step from the Symfony application
directly from GitHub. We have an admin panel and a public site, the idea is
use our InfluxDB PHP SDK to understand how this feature works. We use the
Dependency Injection Container (DiC) provided by Symfony to create our
influxdb.client.</p>
<p>Go into the project’s root and use composer to install the library: composer
require influxdb/influxdb-php The first things to do is add some parameters:
host and port of our InfluxDB. To do that open app/config/parameters.yml and
add this fields:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">influxdb_host</span><span class="pi">:</span> <span class="s">127.0.0.1</span>
<span class="na">influxdb_port</span><span class="pi">:</span> <span class="s">8086</span>
<span class="na">influxdb_db</span><span class="pi">:</span> <span class="s">symfony_influx</span></code></pre></figure>
<p>We use the REST Api to send metrics to InfluxDB, please if your connection
params are different change them.</p>
<p>The second step is configure the Symfony’s
DiC in order to get our client around the application, open
app/config/services.yml and add this line.</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">services</span><span class="pi">:</span>
<span class="c1"># ...</span>
<span class="na">influxdb_client</span><span class="pi">:</span>
<span class="na">class</span><span class="pi">:</span> <span class="s">InfluxDB\Client</span>
<span class="na">arguments</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">%influxdb_host%'</span><span class="pi">,</span> <span class="s1">'</span><span class="s">%influxdb_port%'</span><span class="pi">]</span>
<span class="na">influxdb_database</span><span class="pi">:</span>
<span class="na">class</span><span class="pi">:</span> <span class="s">InfluxDB\Database</span>
<span class="na">arguments</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">%influxdb_db%'</span><span class="pi">,</span> <span class="s1">'</span><span class="s">@influxdb_client'</span><span class="pi">]</span></code></pre></figure>
<p>With this specification we are asking at the DiC to provide a influxdb_client,
it’s a InfluxDB\Client object with two constructor parameters: influxdb_host,
influxdb_port.</p>
<p>InfluxDB could have different databases, influxdb_database is a
service that use the influxdb_clint to work with only one database influxdb_db.
Now we have a influxdb.database ready to be used! Only to try if all works
fine open DefaultController and try to send a page view metrics:</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"> <span class="sd">/**
* @Route("/", name="homepage")
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">indexAction</span><span class="p">(</span><span class="nx">Request</span> <span class="nv">$request</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">get</span><span class="p">(</span><span class="s2">"influxdb_database"</span><span class="p">)</span><span class="o">-&gt;</span><span class="na">writePoints</span><span class="p">([</span><span class="k">new</span> <span class="nx">Point</span><span class="p">(</span>
<span class="s1">'page_view'</span><span class="p">,</span> <span class="c1">// name of the measurement
</span> <span class="mi">1</span> <span class="c1">// the measurement value
</span> <span class="p">)]);</span>
<span class="c1">// replace this example code with whatever you need
</span> <span class="k">return</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">render</span><span class="p">(</span><span class="s1">'default/index.html.twig'</span><span class="p">,</span> <span class="p">[</span>
<span class="s1">'base_dir'</span> <span class="o">=&gt;</span> <span class="nb">realpath</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getParameter</span><span class="p">(</span><span class="s1">'kernel.root_dir'</span><span class="p">)</span><span class="o">.</span><span class="s1">'/..'</span><span class="p">),</span>
<span class="p">]);</span>
<span class="p">}</span></code></pre></figure>
<p><img class="img-fluid" alt="InfluxDB admin panel" src="/img/influxdb_admin.png" /></p>
<p>Go into the homepage and in the meantime do a query like SELECT * FROM
“symfony_influx”.”“.page_view into the InfluxDB’s admin panel, you are sending
a new point after each visit! Very good but we have another target! If you
have some problem and you are using my repository see the difference between
this and the last step on GitHub.</p>
<p>Sent a point in this method it’s not a good
practice because our controller has two responsability: Rendering of the page
Sent a point In this example the situation is not dangerous because the
application is very easy and with a very low traffic, but symfony provide a
strong event system, perfect to split the logic on different classes and
simplify our code, we try to follow this approach for our last step, we create
a listener to sent a point when an user fails a login. In first we must create
a listener into src/AppBundle/Listener/MonitorAuthenticationListener.php.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">AppBundle\Listener</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Symfony\Component\Security\Core\Event\AuthenticationFailureEvent</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">InfluxDB\Point</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">MonitorAuthenticationListener</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$database</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">__construct</span><span class="p">(</span><span class="nv">$database</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-&gt;</span><span class="na">database</span> <span class="o">=</span> <span class="nv">$database</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">onFailure</span><span class="p">(</span><span class="nx">AuthenticationFailureEvent</span> <span class="nv">$event</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-&gt;</span><span class="na">database</span><span class="o">-&gt;</span><span class="na">writePoints</span><span class="p">([</span><span class="k">new</span> <span class="nx">Point</span><span class="p">(</span>
<span class="s1">'login'</span><span class="p">,</span>
<span class="mi">1</span><span class="p">,</span>
<span class="p">[</span><span class="s1">'status'</span> <span class="o">=&gt;</span> <span class="s1">'error'</span><span class="p">]</span>
<span class="p">)]);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>We use the DiC to attach this listener at the security.authentication.failure
event. This event is called after each failed login. To do that open
app/config/services.yml and add this configuration.</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">services</span><span class="pi">:</span>
<span class="c1"># ....</span>
<span class="s">security.authentication.monitoring</span><span class="pi">:</span>
<span class="na">class</span><span class="pi">:</span> <span class="s">AppBundle\Listener\MonitorAuthenticationListener</span>
<span class="na">arguments</span><span class="pi">:</span> <span class="pi">[</span><span class="s1">'</span><span class="s">@influxdb_database'</span><span class="pi">]</span>
<span class="na">tags</span><span class="pi">:</span>
<span class="pi">-</span> <span class="pi">{</span> <span class="nv">name</span><span class="pi">:</span> <span class="nv">kernel.event_listener</span><span class="pi">,</span> <span class="nv">event</span><span class="pi">:</span> <span class="nv">security.authentication.failure</span><span class="pi">,</span> <span class="nv">method</span><span class="pi">:</span> <span class="nv">onFailure</span> <span class="pi">}</span></code></pre></figure>
<p>We are injecting into the constructor our influxdb database, in this way we use
it to send points like our old example into the controller. This is the last
practical section of this tutorial, please if you have lost something try to
check this diff from the last step on GitHub. Try to do some wrong login and
check the situation into the Admin Panel with a query like</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="nv">"symfony_influx"</span><span class="p">.</span><span class="nv">""</span><span class="p">.</span><span class="n">login</span><span class="p">.</span></code></pre></figure>
<p><img class="img-fluid" alt="InfluxDB admin panel" src="/img/chronograf.png" /></p>
<p>The admin panel is not the best way to check our metrics, InfluxData provide a
great dashboard called Chronograf, try to use this metric to create a graph
specific to understand how your feature works. This post is only a getting
started to understand a good way to send metrics without connect directly your
business logic with the monitoring system, but with a real traffic this
approach is totally inefficient.</p>
<p>Send point by point increase the traffic in your network and the latency create
performance problems, telegraf is a collector that you can use to mitigate this
problem, in this way you can not send your points directly to InfluxDB but you
can use this agent installed on your server that collect and send bulk of data
for you.</p>
Docker 1.12 orchestration built-in2016-06-20T10:08:27+00:00http://gianarb.it/blog/docker-1-12-orchestration-built-in<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Some
tests with Docker 1.12! <a href="https://t.co/budUOtMuBB">https://t.co/budUOtMuBB</a> <a href="https://twitter.com/hashtag/docker?src=hash">#docker</a> <a href="https://twitter.com/hashtag/DockerCon?src=hash">#DockerCon</a>
orchestration, swarm and services.</p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/744977855277309953">June 20,
2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>During the DockerCon 2016 docker announced Docker 1.12 release.
One of the news stories around the new version is the orchestration system built directly
inside the engine, this feature allow us to use swarm
without installing it separately from outside, it’s now a feature provided by Docker directly.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>Now we have a new set of commands that allow us to orchestrate containers
across a cluster.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker swarm
docker node
docker service</code></pre></figure>
<p>All these commands are focused on increasing our ability to orchestrate our
containers and also join them in services.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">#!/bin/bash</span>
<span class="c"># create swarm manager</span>
docker-machine create <span class="nt">-d</span> virtualbox sw1
<span class="nb">echo</span> <span class="s2">"sudo /etc/init.d/docker stop &amp;&amp; </span><span class="se">\</span><span class="s2">
curl https://test.docker.com/builds/Linux/x86_64/docker-1.12.0-rc2.tgz | </span><span class="se">\</span><span class="s2">
tar xzf - &amp;&amp; sudo mv docker/* /usr/local/bin &amp;&amp; </span><span class="se">\</span><span class="s2">
rm -rf docker/ &amp;&amp; sudo /etc/init.d/docker start"</span> | <span class="se">\</span>
docker-machine ssh sw1 sh -
docker-machine ssh sw1 docker swarm init
<span class="c"># create another swarm node</span>
docker-machine create <span class="nt">-d</span> virtualbox sw2
<span class="nb">echo</span> <span class="s2">"sudo /etc/init.d/docker stop &amp;&amp; </span><span class="se">\</span><span class="s2">
curl https://test.docker.com/builds/Linux/x86_64/docker-1.12.0-rc2.tgz | </span><span class="se">\</span><span class="s2">
tar xzf - &amp;&amp; sudo mv docker/* /usr/local/bin &amp;&amp; </span><span class="se">\</span><span class="s2">
rm -rf docker/ &amp;&amp; sudo /etc/init.d/docker start"</span> | <span class="se">\</span>
docker-machine ssh sw2 sh -
docker-machine ssh sw2 docker swarm join <span class="k">$(</span>docker-machine ip sw1<span class="k">)</span>:2377</code></pre></figure>
<p>another Captain wrote this script that I just updated to work
with the public Docker 1.12-rc2. We can use this script to create a cluster with
virtual box ready to be used. After this script you can see the number of
workers and masters, in this case your one and one.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker node <span class="nb">ls</span></code></pre></figure>
<p>Docker 1.12 has a built-in set of primitive functions to orchestrate your containers just
like a summary. The main commands that you must run to create a cluster are</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c">## On the master to start your cluster</span>
<span class="nv">$ </span>docker swarm init <span class="nt">--listen-addr</span> &lt;master-IP<span class="o">(</span>this ip<span class="o">)&gt;</span>:2377
<span class="c">## on each node to add it into the cluster</span>
<span class="nv">$ </span>docker swarm join &lt;master-ip&gt;:2377</code></pre></figure>
<p><img class="img-fluid" alt="Docker Swarm architecture" src="/img/posts/swarm_arch.png" /></p>
<p>If you are not confident with docker swarm this is the architecture, this graph
is provided by Docker Inc. and explains really well the design around this project.
The principal actors are managers and workers, managers are the brains of the
system, they dispatch schedules and remember services and containers. Workers
execute these commands.</p>
<p>The cluster is secure because each node has a proper TLS
identity and all communications are encrypted end to end by default with a
automatic key rotation in order to increase the security around the keys use in
the cluster.</p>
<p><a href="https://raft.github.io/">Raft</a> is the consensual protocol used to distribute
message around the cluster and check the number of nodes, it’s complex
algorithm but really interested I have in plan another article about it but the
offical site contains a lot of details about it.</p>
<p>We already saw the concept of services in docker-compose they are a single or a
group of containers to describe your ecosystem, you can scale a specific
service or orchestrate it across your cluster. It’s the same here, you don’t have
a specification file like compose at the moment but anyway you can run a bunch
of commands to create your service.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ </span>docker service create <span class="nt">--name</span> helloworld <span class="nt">--replicas</span> 1 alpine ping docker.com</code></pre></figure>
<p>With this example we push up a new service helloworld. It has one container from
the alpine image and it pings docker.com site.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker service <span class="nb">ls</span></code></pre></figure>
<p>To watch all our services, we can also inspect a service</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker service inspect &lt;container_id&gt;</code></pre></figure>
<p>There is a new concept, when you run a service you are also creating a task,
this task represents the container/s under your service, in this case we have
just one task</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker service tasks helloworld</code></pre></figure>
<p>When you scale your service you are creating new tasks</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker service scale <span class="nv">helloworld</span><span class="o">=</span>10</code></pre></figure>
<p>Now you can see 10 tasks that are running and you can inspect one of them,
inside you can find the containerId and you can, for example, follow logs</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">22:17 <span class="nv">$ </span>docker inspect 6fhfse4it8lwzlsk1t5sd5jbk
<span class="o">[</span>
<span class="o">{</span>
<span class="s2">"ID"</span>: <span class="s2">"6fhfse4it8lwzlsk1t5sd5jbk"</span>,
<span class="s2">"Version"</span>: <span class="o">{</span>
<span class="s2">"Index"</span>: 67
<span class="o">}</span>,
<span class="s2">"CreatedAt"</span>: <span class="s2">"2016-06-18T21:06:36.707664178Z"</span>,
<span class="s2">"UpdatedAt"</span>: <span class="s2">"2016-06-18T21:06:39.241942781Z"</span>,
<span class="s2">"Spec"</span>: <span class="o">{</span>
<span class="s2">"ContainerSpec"</span>: <span class="o">{</span>
<span class="s2">"Image"</span>: <span class="s2">"alpine"</span>,
<span class="s2">"Args"</span>: <span class="o">[</span>
<span class="s2">"ping"</span>,
<span class="s2">"docker.com"</span>
<span class="o">]</span>
<span class="o">}</span>,
<span class="s2">"Resources"</span>: <span class="o">{</span>
<span class="s2">"Limits"</span>: <span class="o">{}</span>,
<span class="s2">"Reservations"</span>: <span class="o">{}</span>
<span class="o">}</span>,
<span class="s2">"RestartPolicy"</span>: <span class="o">{</span>
<span class="s2">"Condition"</span>: <span class="s2">"any"</span>,
<span class="s2">"MaxAttempts"</span>: 0
<span class="o">}</span>,
<span class="s2">"Placement"</span>: <span class="o">{}</span>
<span class="o">}</span>,
<span class="s2">"ServiceID"</span>: <span class="s2">"24e0pojscuj2irvlxvx2baiid"</span>,
<span class="s2">"Slot"</span>: 2,
<span class="s2">"NodeID"</span>: <span class="s2">"55v4jjzf56mcwnhbwvn4cq1rs"</span>,
<span class="s2">"Status"</span>: <span class="o">{</span>
<span class="s2">"Timestamp"</span>: <span class="s2">"2016-06-18T21:06:36.7110425Z"</span>,
<span class="s2">"State"</span>: <span class="s2">"running"</span>,
<span class="s2">"Message"</span>: <span class="s2">"started"</span>,
<span class="s2">"ContainerStatus"</span>: <span class="o">{</span>
<span class="s2">"ContainerID"</span>: <span class="s2">"4ec69142e3e886098915140663737f4176c6de5afe9f2fad1f5b2439d8fc336d"</span>,
<span class="s2">"PID"</span>: 3627
<span class="o">}</span>
<span class="o">}</span>,
<span class="s2">"DesiredState"</span>: <span class="s2">"running"</span>
<span class="o">}</span>
<span class="o">]</span>
22:17 <span class="nv">$ </span>docker logs <span class="nt">-f</span> 6fhfse4it8lwzlsk1t5sd5jbk</code></pre></figure>
<p>At this point it is a normal container and it’s running on your cluster.
Well I tried to explain the main concept around this big feature provided by
Docker 1.12, the last example is just to cover the DNS topic.</p>
<p>I created an application that serve an http server and print the current IP.
Each server has an internal load balancer that dispatches traffic in round robin
between the different tasks.
In this way it’s totally transparent, you can just
resolve your service with a normal URL, docker will do the rest for you.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$.</span> docker service create —-name micro —-replicas 10 —-publish 8000/tcp gianarb/micro</code></pre></figure>
<p><a href="https://github.com/gianarb/micro">Micro</a> is an application that exposes an
http server on port 8000 and print the current ip, now we have 10 tasks with
this service.
To grab the current entry point for our service we can inspect it
and search for this information:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$.</span> docker service inspect &lt;id-service&gt;
...
<span class="s2">"Endpoint"</span>: <span class="o">{</span>
<span class="s2">"Spec"</span>: <span class="o">{}</span>,
<span class="s2">"Ports"</span>: <span class="o">[</span>
<span class="o">{</span>
<span class="s2">"Protocol"</span>: <span class="s2">"tcp"</span>,
<span class="s2">"TargetPort"</span>: 8000,
<span class="s2">"PublishedPort"</span>: 30000
<span class="o">}</span>
<span class="o">]</span>,
<span class="s2">"VirtualIPs"</span>: <span class="o">[</span>
<span class="o">{</span>
<span class="s2">"NetworkID"</span>: <span class="s2">"890fivvc6od3pa4rxd281lobb"</span>,
<span class="s2">"Addr"</span>: <span class="s2">"10.255.0.5/16"</span>
<span class="o">}</span>
<span class="o">]</span>
<span class="o">}</span>
...</code></pre></figure>
<p>In this case our published port is 3000, we can call <ip>:3000 to resolve our
service, if you try to do multi request you can see your IP chances because the
internal DNS is calling different containers.</ip></p>
<p>This is just an overview about the features but there are other powerful news
like DAB, stacks and how do an easy update of your containers, this could be
the topic around my next article. Please stay in touch follow me on
<a href="https://github.com/gianarb">Twitter</a> to chat and receive news about the next articles.</p>
<blockquote>
<p>Thanks <a href="https://twitter.com/gpelly">@gpelly</a> for your review!</p>
</blockquote>
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=90cae01e86" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">News from Docker Planet</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_90cae01e86" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
<div class="row">
<div class="col-md-12">
<a href="//gianarb.it/planet/docker.html"><b>Docker Planet</b></a> is a collection of posts, events and other
information about Docker and all the container ecosystem. Read other
articles and stay in touch, subsbrite to the newsletter.
</div>
</div>
A little bit of refactoring2016-04-24T10:08:27+00:00http://gianarb.it/blog/a-little-bit-of-refactoring<p>I wrote this note during the <a href="http://phpkonf.org/">PHPKonf</a>, I spoke about
<a href="http://gianarb.it/jenkins-real-world/#/">Jenkins and Continuous Delivery</a> but
during the days I followed few interested talks and this is my list of notes.</p>
<p>Is the code reusable? For a few people the response is yes, for other ones is
no. I agree with <a href="https://twitter.com/ocramius">Ocramius</a> that the response is
no. An abstraction is reusable, an interface is reusable, but it’s very hard to
reuse a final implementation. First of all because when you finish to write it
your code is already old, your function is already legacy and you start to
maintain it, you search bugs and edge cases.</p>
<p>One of the way to reduce the time dedicated to do refactroring is prevent and
defende your code from bad integration, in OOP usually you have a sort of
visibility (private, public, protected) and other different way to defend your
code, opinable but the ocramius’s talk about <a href="https://ocramius.github.io/extremely-defensive-php/#/">Extremly defensive
php</a> is good to see.</p>
<p>Refactoring is a methodology to make better your code. There are different
improvement topics like readable, performance, solidity.</p>
<ul>
<li>make your code readable for the new generation is one of the best stuff that
you can do to show your love for your team and your company.</li>
<li>If your site require more time to be loaded usually you lost your client.
“less performance is a bug” cit. <a href="https://twitter.com/fabpot">Fabien Potencier</a></li>
<li>When your run your code all it’s fine, you are a good developer and your
feature works. After the deploy in production your code is the same but
usually there is a bd category of people, your client, that will use it in a
very strange way, usually it’s synonym of bug or edge case. Each bug fix make
your code more solid.</li>
</ul>
<p>Test your code before start to change it, you know automation is good but if
you love seems a machine to it manually. Setup a continuous integration
system, it can be do just one step like run tests but remember to increase that
with all steps that you usually do to test the compliance of your code like
style, standard, static analysis just to enforce that you are not a machine.
Create a good environment and an automatic lifecycle for your application allow
you to stay focused on the code and not lost your time around stupid task,
remember that when a routine is good the machine fail less respect a human,
usually.</p>
<p>Refactoring is one of the best stuff that you can do for other people and to
make your feature ready for the real world, usually it’s hard for no-tech
company understands it because few times they don’t see any kind of change
create a good environment to save time and use it to do refactoring is a godo
strategy. Automation is the unique method that I know to do that. There are
different layer of automation just to start my 2coins is just put a make file
on your codebase and when you do something for the second time stop to write it
on your console and write a new make task to share with you team. After that
install Jenkins and allow it do run this task for you before put on your on the
master branch (for git users, trunk for svn users).</p>
<p>Make your development environment comfortable and increase the conformability’s
perception about the lifecycle it’s the best way to do refactoring without the
fear to die. If you are fear to die usually you don’t do nothing.</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Great
talk as always Gianluca :) <a href="https://twitter.com/GianArb">@GianArb</a>
<a href="https://twitter.com/hashtag/phpkonf?src=hash">#phpkonf</a> <a href="https://t.co/ZW2G1UsXm7">pic.twitter.com/ZW2G1UsXm7</a></p>&mdash;
Fontana Lorenzo (@fntlnz) <a href="https://twitter.com/fntlnz/status/733986655334486016">May 21,
2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Add the PHPKonf in your list! See you next year!</p>
Docker inside docker and overview about Jenkins 22016-04-01T10:08:27+00:00http://gianarb.it/blog/docker-inside-docker-and-jenkins-2<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/hashtag/docker?src=hash">#docker</a> inside docker
and an overview about Jenkins 2 <a href="https://t.co/qa5ddjfhrs">https://t.co/qa5ddjfhrs</a> <a href="https://twitter.com/docker">@docker</a> <a href="https://twitter.com/jenkinsci">@jenkinsci</a> <a href="https://twitter.com/hashtag/container?src=hash">#container</a></p>&mdash;
Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/727876226875068416">May 4,
2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Jenkins is one of the most famous
continuous integration and deployment tools, it’s written in Java and it helps
you to manage your pipeline and all tasks that help you to put your code in
production or manage your build.</p>
<p>The announcement of Jenkins release of version 2 few days ago, is one of the
best release of this year in my opinion.</p>
<p>The previous version is very stable but it has a lot of years and the ecosystem
is totally different. I am happy to see a strong refurbishment to get the best
of this powerful tool with a series of new feature like:</p>
<ul>
<li>Nice installation wizard</li>
<li>Refactoring of the design, one of the most critical
feature of the previous version</li>
<li>Good and modern set of plugins like <a href="https://jenkins.io/solutions/pipeline/">Jenkins
Pipeline</a> to manage your build</li>
</ul>
<p>Jenkins is truly a wonder but the tool of the moment it’s docker, engine
that allow you to work easier with the containers.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>This two tools together are perfect to create an isolated environment to test
and deploy your applications.</p>
<p>The first setup could be install Jenkins on your
server and use a plugin to manage the integration and trigger your test inside
an isolated environment, the container.</p>
<p>Great work but in my opinion reproducibility is one of the critical point when
you deal with plugins if you can not run your build on your local environment
easily then you have a problem. Secondly if the container could be a good
solution to deploy and maintain a solid and isolated application, why your
Jenkins has not the privilege to run inside a container? In this perspective
how can we run container inside a container?</p>
<p>Ok, now its the time to figure it out how to solve the problems.</p>
<p>We can use the official Jenkins image to put jenkins inside a container, but I
worked on my personal alpine installation, light and easy, <a href="https://github.com/gianarb/dockerfile/blob/master/jenkins/2.0/Dockerfile">here is the
dockerfile</a>
and we can pull it:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker pull gianarb/jenkins:2.0</code></pre></figure>
<p>If you are interested the main article to understand how run docker inside
docker is written by
<a href="https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/">jpetazzo</a>,
the idea is run our jenkins container with <code class="highlighter-rouge">-privileged</code> enabled and share our
docker binary and the socket <code class="highlighter-rouge">/var/run/docker.sock</code> to manage our
communications.</p>
<ul>
<li><code class="highlighter-rouge">/var/run/docker.sock</code> is the entrypoint of the docker daemon</li>
<li><code class="highlighter-rouge">docker</code> the command is like a client that sends commands to socket</li>
<li><code class="highlighter-rouge">--privileged</code> give extended privileges to our container</li>
</ul>
<p>Translated in a docker command:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker run <span class="nt">-v</span> /var/run/docker.sock:/var/run/docker.sock <span class="se">\</span>
<span class="nt">-v</span> <span class="k">$(</span>which docker<span class="k">)</span>:/usr/local/bin/docker <span class="se">\</span>
<span class="nt">-p</span> 5000:5000 <span class="nt">-p</span> 8080:8080 <span class="se">\</span>
<span class="nt">-v</span> /data/jenkins:/var/jenkins <span class="se">\</span>
<span class="nt">--privileged</span> <span class="se">\</span>
<span class="nt">--restart</span> always <span class="se">\</span>
gianarb/jenkins:2.0</code></pre></figure>
<p>We connect on <code class="highlighter-rouge">http://docker-ip:8080</code> and start the new awesome wizard!</p>
<p><img class="img-fluid" alt="First Jenkins 2 page, grab from the log your key and start" src="/img/docker-in-docker/jenkins2-start.png" /></p>
<p><img class="img-fluid" alt="Jenkins's plugins wizard" src="/img/docker-in-docker/jenkins2-plugin.png" /></p>
<p>To verify that all work we can create a new job it only runs <code class="highlighter-rouge">docker ps -a</code> our
expectation is the same list of containers that we have out of jenkins.</p>
<p><img class="img-fluid" alt="Result of the first build" src="/img/docker-in-docker/jenkins2-result.png" /></p>
<p>Now we can use run command from jenkins to manage our build with docker without
any kind of plugins but anyway you are free to use <a href="https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin">Docker
Plugin</a> to start
your build.</p>
<p>I used Jenkins like an example to run docker inside another container but you
can use the same strategy to do the same with your applications if they require
a strong connection with docker.</p>
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=90cae01e86" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">News from Docker Planet</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_90cae01e86" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
<div class="row">
<div class="col-md-12">
<a href="//gianarb.it/planet/docker.html"><b>Docker Planet</b></a> is a collection of posts, events and other
information about Docker and all the container ecosystem. Read other
articles and stay in touch, subsbrite to the newsletter.
</div>
</div>
Happy docker's birthday and thanks2016-03-25T10:08:27+00:00http://gianarb.it/blog/happy-docker-bday-and-thanks<p>Just a post to say thanks docker for your awesome community and happy birthday!</p>
<p>This week is the “Docker’s birthday week” and already it this amazing, one week
of birthday, a lot of MeetUp groups this week done a Tutorial Meetup to help
people to start with Docker, Dublin made it very well!</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Fantastic turnout for the <a href="https://twitter.com/hashtag/dockerbday?src=hash">#dockerbday</a> <a href="https://twitter.com/Workday">@workday</a>. Thanks to everyone who
attended and completed the voting app!! <a href="https://t.co/zWQssvyHSd">pic.twitter.com/zWQssvyHSd</a></p>&mdash;
TomWillFixIT (@tomwillfixit) <a href="https://twitter.com/tomwillfixit/status/712749765151297537">March 23,
2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>50 people to understand as docker and all ecosystem works and to eat a slice of cake (thanks WorkDay)</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">All
it&#39;s ready.. We can go! <a href="https://twitter.com/hashtag/dockerbday?src=hash">#dockerbday</a> Dublin..
Some problem? Don&#39;t worry to ask! <a href="https://t.co/9qz3V9mW9y">pic.twitter.com/9qz3V9mW9y</a></p>&mdash;
Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/712705450786099200">March 23,
2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>There are different kind of developers, I am happy to work and to follow each
communities that provide a good tools and increase the quality of my work, I
spend much time accross different team like doctrine, InfluxDB and I am very
happy to see as Docker make a big effort to involve and to use its community.</p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>I wase member of the beautiful mentor team (we will share a “retro pic” next month
because we forget to do it) and I am happy to see that we done a good work.</p>
<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/GianArb">@GianArb</a> tnks 4 the help 2nite
Gianluca!! Appreciate it <a href="https://twitter.com/hashtag/DublinDocker?src=hash">#DublinDocker</a> <a href="https://twitter.com/hashtag/dockerbday?src=hash">#dockerbday</a></p>&mdash;
Delpedro (@Delpedro47) <a href="https://twitter.com/Delpedro47/status/712745923848351744">March 23,
2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Today we seen that also che community appreciate it!
Happy Birthday and <a href="http://www.meetup.com/Docker-Dublin/">see you next month</a>!</p>
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=90cae01e86" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">News from Docker Planet</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_90cae01e86" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
<div class="row">
<div class="col-md-12">
<a href="//gianarb.it/planet/docker.html"><b>Docker Planet</b></a> is a collection of posts, events and other
information about Docker and all the container ecosystem. Read other
articles and stay in touch, subsbrite to the newsletter.
</div>
</div>
Some days of work vs Jenkins CI2016-02-21T10:08:27+00:00http://gianarb.it/blog/some-days-of-work-vs-jenkins-ci<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr"><a href="https://t.co/02HbnkzRsS">https://t.co/02HbnkzRsS</a> &quot;Some Days of work vs <a href="https://twitter.com/hashtag/JenkinsCI?src=hash">#JenkinsCI</a>&quot; Little things about continuous integration <a href="https://twitter.com/hashtag/ci?src=hash">#ci</a> <a href="https://twitter.com/hashtag/dev?src=hash">#dev</a></p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/709466156453732352">March 14, 2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>Guys please move down your hands, I love JenkinsCI! I am not here to write a
bad post about it!
I am here to share few days of reasonings about continuous
inteagrations, Jenkins CI and all this strong topic.</p>
<h2 id="reproducible">Reproducible</h2>
<p>There are a lot of tools that you can use to run tasks,
ant, make, grunt. Use it to run section or all your build on your local
environment, this appraoch increase the value your tasks becuase you use and
test them more times.
To have a reproducible build help you to maintain splitted your flow by your
runner maybe Jenkins is perfect but the are other tools and services:
Travis-CI, CircleCI, Drone, don’t create a big dependency with your environment.</p>
<h2 id="speedy">Speedy</h2>
<p>A slow test suite is a bad idea, in 1 minutes I can maintain the
focus on the execution but 5 minutes are a lot, you can take a coffé or
start to think about another task and return on your old task require an
effort. This several focus switch it’s not good, and at the same time you
lost 5 minutes for each build and for every engineer after 1 week are a lot
of money.</p>
<h2 id="versionable">Versionable</h2>
<p>I lost more time about this point and I am not sure if it is a required point
or not but TravisCI for example use a yml specification file, this file doesn’t
describe only your build but it part of the story of your application, if you
include it into the VCS. Could be it a value for your pipeline?</p>
<h2 id="maintainable">Maintainable</h2>
<p>There are a lot of tools that you can choice to create the perfect pipeline ant
it’s very easy lost your focus and start to use too much tools, you must try
all but it’s your task to create the perfect sub-set of tools the point 1
(Reusability) increase the value, use tools that you can reuse during the daily
work of your team to increase the develop and go the flow better.
Each tools that you add seems perfect until they don’t becase a problem.</p>
<h2 id="scalable">Scalable</h2>
<p>An easy way to decrease the time for your job is split it in different little
jobs and run them in parallel, you can check the codestyle and run your test
suite in the same time for example.
Another good reason to create a scalable environment for your jobs is because
your company would grow and the continuous integration system burns to helps it
to grow and not to stop it.</p>
<h2 id="unique">Unique</h2>
<p>Jenkins, vagrant, ant, make, drone, docker are only a list of amazing tools to
create the perfect pipeline to deploy and test your code but they are only a
means the goal is indeed the best pipeline for your code and for your team.
Observe how your team works, which the requirments and criticalities and design
the best pipeline for your use case.</p>
<h2 id="communication-layer">Communication layer</h2>
<p>One goal for your team is understand the status of the build without logged in
any application, because enter into the Jenkins site (at first because it is
not beautiful :P ) and it is another step to do other: create feature branch,
submit pull request, write code lalala..
Use directly the pull request to create a connection with your job, you
continuous integration system can submit a new comment or if you are working
with GitHub you can use the status check, in this way you can help your
colleagues during them work and remove a jump.</p>
<p>With JenkinsCi you can do all but if you lost more time to create your best
pipeline? Maybe you don’t know it or maybe it is not the best tool for your use
case. Jenkins is flexible, but the flexibility is only the number of plugins
that you can install?</p>
<p>I don’t know I use it but I am happy to experiment and there are a lot of new
technologies and tools that maybe can help us to do a good work, with or
without JenkinsCI.</p>
<h2 id="as-a-microservices">As a microservices</h2>
<p><img src="/img/pipeline.svg" alt="Continuous Integration and Deploy pipeline" /></p>
<p>This is a summary of a pipeline, each pipeline follows this steps and from this
point of view seems very easy!
Jenkins, drone as very stong solution but they are all in one, if you follow
this image it’s clear that maybe to create the own pipeline for your projects
play with the LEGO to mount the best steps for your team and for your project
it’s possible.</p>
<p>I am happy to share some projects to implement this approach.</p>
<h2 id="slimmer-proof-of-concept">Slimmer, proof of concept</h2>
<p>I tried to create a runner for my test suite, <a href="https://github.com/gianarb/slimmer">slimmer</a>,
to implement this thought with docker and go.
Go offers a lot of libraries and tools to create something in a bit of time and
docker it’s perfect because it creates isolated environment and it’s very easy
to scale with Swarm.
In practice at the moment this console app exec a <code class="highlighter-rouge">build.slimmer</code> a bash script
executable flexible and versionable.
<a href="https://travis-ci.org">TravisCI</a> is powerful but the YML file is it a good way
to describe a build? It’s flexible? Maybe yes but I am curious to try a “low
level” approach, because finally all becames a series of commands.
I created also a series of agent to trigger notification quicly:
<a href="https://github.com/gianarb/ircer">ircer</a>,
<a href="https://github.com/gianarb/slacker">slacker</a>. You can use them to notify the
result of your build.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">composer install
vendor/bin/phpunit
<span class="nv">RESULT</span><span class="o">=</span><span class="nv">$?</span>
curl <span class="nt">-LSs</span> https://github.com/gianarb/ircer/releases/download/0.1.0/ircer_0.1.0_linux_386 <span class="o">&gt;</span> ircer
chmod 755 ircer
<span class="k">if</span> <span class="o">[</span> <span class="nv">$RESULT</span> <span class="o">=</span> 0 <span class="o">]</span><span class="p">;</span> <span class="k">then</span>
./ircer <span class="nt">-j</span> tech-team <span class="nt">-m</span> <span class="s2">"You are a great develop. Your build works"</span>
<span class="k">else</span>
./ircer <span class="nt">-j</span> tech-team <span class="nt">-m</span> <span class="s2">"No bad but your build doesn't work"</span>
<span class="k">fi</span></code></pre></figure>
<p>This is an example of <code class="highlighter-rouge">build.slimmer</code> with an IRC notification, it is a PoC and
I prepared a little <a href="http://gianarb.it/slimmer-poc-slide/#/">presentation</a> to
receive some feedback and I presented it during a Dublin Go Meetup</p>
<div class="row">
<div class="col-md-12 text-center">
<iframe width="560" height="315" src="https://www.youtube.com/embed/CWCHT3GClMM" frameborder="0" allowfullscreen=""></iframe>
</div>
</div>
<p>I wait some feedback if you are interested about continuous integration and
continuous delivery.</p>
ChatOps create your IRC bot in Go2016-02-21T10:08:27+00:00http://gianarb.it/blog/chatops-create-your-own-irc-bot-in-go<p>The infrastructure as a service (IaaS) opened new ways to manage your
infrastructure.
Use an API to create, destroy and update your virtual machine
is one of the biggest revolutions of our sector.</p>
<p>A lot of companies and a lot of DevOps started to create own assistence to
increase the automation or to check the status of them infrastructure, in top of
all GitHub provided a series of awesome blogpost and tools to describe this
approach that it has a name: ChatOps.</p>
<ul>
<li><a href="https://hubot.github.com/">HuBot</a> is a beautiful tools written in node.js to provide smart bot.</li>
<li><a href="https://www.pagerduty.com/blog/what-is-chatops/">So, What is ChatOps? And How do I Get Started?</a> by PagerDuty</li>
<li><a href="https://github.com/blog/968-say-hello-to-hubot">Say Hello to Hubot</a> by GitHub</li>
</ul>
<div class="row">
<div class="col-md-12 text-center">
<iframe width="560" height="315" src="https://www.youtube.com/embed/IhzxnY7FIvg" frameborder="0" allowfullscreen=""></iframe>
</div>
</div>
<p>IRC is an application layer protocol that facilitates communication. One of the
most famouse open IRC server is freenode all most important open source projects
use it to chat.</p>
<p>This concept is already applyed it because most projects are your personal bot,
for example Zend use Zend\Bot a good assistence written by DASPRiD.</p>
<p>The ChatOps is an assistence oriented to decrease the distance between your
infrastacture and your communication channels.</p>
<p>I wrote a low level library to communicate on IRC protocol, we can try to use it to
write our dummy bot.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="k">package</span><span class="x"> </span><span class="n">main</span><span class="x">
</span><span class="k">import</span><span class="x"> </span><span class="p">(</span><span class="x">
</span><span class="s">"log"</span><span class="x">
</span><span class="s">"fmt"</span><span class="x">
</span><span class="s">"regexp"</span><span class="x">
</span><span class="s">"bufio"</span><span class="x">
</span><span class="s">"net/textproto"</span><span class="x">
</span><span class="s">"github.com/gianarb/go-irc"</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">main</span><span class="p">(){</span><span class="x">
</span><span class="n">secretary</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">NewBot</span><span class="p">(</span><span class="x">
</span><span class="s">"irc.freenode.net"</span><span class="p">,</span><span class="x">
</span><span class="s">"6667"</span><span class="p">,</span><span class="x">
</span><span class="s">"SybilBot"</span><span class="p">,</span><span class="x">
</span><span class="s">"SybilBot"</span><span class="p">,</span><span class="x">
</span><span class="s">"#channel-name"</span><span class="p">,</span><span class="x">
</span><span class="s">""</span><span class="p">,</span><span class="x">
</span><span class="p">)</span><span class="x">
</span><span class="n">conn</span><span class="p">,</span><span class="x"> </span><span class="n">_</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">secretary</span><span class="o">.</span><span class="n">Connect</span><span class="p">()</span><span class="x">
</span><span class="k">defer</span><span class="x"> </span><span class="n">conn</span><span class="o">.</span><span class="n">Close</span><span class="p">()</span><span class="x">
</span><span class="n">reader</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">bufio</span><span class="o">.</span><span class="n">NewReader</span><span class="p">(</span><span class="n">bot</span><span class="o">.</span><span class="n">conn</span><span class="p">)</span><span class="x">
</span><span class="n">tp</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">textproto</span><span class="o">.</span><span class="n">NewReader</span><span class="p">(</span><span class="n">reader</span><span class="p">)</span><span class="x">
</span><span class="k">for</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">line</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">tp</span><span class="o">.</span><span class="n">ReadLine</span><span class="p">()</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">log</span><span class="o">.</span><span class="n">Fatal</span><span class="p">(</span><span class="s">"unable to connect to IRC server "</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">isPing</span><span class="p">,</span><span class="x"> </span><span class="n">_</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">regexp</span><span class="o">.</span><span class="n">MatchString</span><span class="p">(</span><span class="s">"PING"</span><span class="p">,</span><span class="x"> </span><span class="n">line</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">isPing</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="no">true</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="n">bot</span><span class="o">.</span><span class="n">Send</span><span class="p">(</span><span class="s">"PONG"</span><span class="p">);</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">fmt</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span><span class="x"> </span><span class="n">line</span><span class="p">)</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="p">}</span></code></pre></figure>
<p>With this code you have a bot, in this case her name is SybilBot and at the
moment it suppot only the PING PONG flow, without this helth system your bot go
down after few time.</p>
<p>You can use the same log to add other actions</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">yourAction</span><span class="p">,</span><span class="x"> </span><span class="n">_</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">regexp</span><span class="o">.</span><span class="n">MatchString</span><span class="p">(</span><span class="s">"CheckSomething"</span><span class="p">,</span><span class="x"> </span><span class="n">line</span><span class="p">)</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">yourAction</span><span class="x"> </span><span class="o">==</span><span class="x"> </span><span class="no">true</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="c">// Do Something</span><span class="x">
</span><span class="p">}</span></code></pre></figure>
<p><a href="https://github.com/gianarb/go-irc">go-irc</a> allow you to communicate over IRC protocol, our but is very stupid I
like the idea! If you are working on this topic, in go or in other language
please ping me! I am very happy to know your bot!</p>
InfluxDB PHP 1.3.0 is ready to go2016-02-18T10:08:27+00:00http://gianarb.it/blog/influxdb-php-release-1-3-0<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">Shout out to <a href="https://twitter.com/GianArb">@GianArb</a> for shipping a new release of the InfluxDB-PHP library! Here&#39;s what&#39;s new: <a href="https://t.co/tJQIu9OCbL">https://t.co/tJQIu9OCbL</a></p>&mdash; InfluxData (@InfluxDB) <a href="https://twitter.com/InfluxDB/status/704403294592970752">February 29, 2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>We are happy to annuonce a new minor release, <a href="https://github.com/influxdata/influxdb-php">Influxdb-php library</a> 1.3.0.</p>
<p>This is a list of PRs merged in 1.3.0 since 1.2.2:</p>
<ul>
<li><a href="https://github.com/influxdata/influxdb-php/pull/36">#36</a> Added quoting of dbname in queries</li>
<li><a href="https://github.com/influxdata/influxdb-php/pull/35">#35</a> Added orderBy to query builder</li>
<li><a href="https://github.com/influxdata/influxdb-php/pull/37">#37</a> Fixed wrong orderby tests</li>
<li><a href="https://github.com/influxdata/influxdb-php/pull/38">#38</a> Travis container-infra and php 7</li>
</ul>
<p>The <code class="highlighter-rouge">QueryBuilder</code> now support the orderBy function to order our data, InfluxDB supports it from version 0.9.4.</p>
<figure class="highlight"><pre><code class="language-sql" data-lang="sql"><span class="k">select</span> <span class="o">*</span> <span class="k">from</span> <span class="n">cpu</span> <span class="k">order</span> <span class="k">by</span> <span class="n">value</span> <span class="k">desc</span></code></pre></figure>
<p>Now you can do it in PHP</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">database</span><span class="o">-&gt;</span><span class="na">getQueryBuilder</span><span class="p">()</span>
<span class="o">-&gt;</span><span class="na">from</span><span class="p">(</span><span class="s1">'cpu'</span><span class="p">)</span>
<span class="o">-&gt;</span><span class="na">orderBy</span><span class="p">(</span><span class="s1">'value'</span><span class="p">,</span> <span class="s1">'DESC'</span><span class="p">)</span><span class="o">-&gt;</span><span class="na">getQuery</span><span class="p">();</span></code></pre></figure>
<p>We are increase our Continuous Integration system in order to check our code with PHP7, it’s perfect!</p>
<p>We escape our query to support reserved keyword like <code class="highlighter-rouge">database</code>, <code class="highlighter-rouge">servers</code> personally I prefer avoid this type of word but you are free to use them.</p>
<p>Please we are very happy to understand as the PHP community use this library and InfluxDB, please share your experience and your problem into the repository, on IRC (join influxdb on freenode) and we wait you on <a href="https://twitter.com/influxdata">Twitter</a>.</p>
<p>Remeber to update your <code class="highlighter-rouge">composer.json</code>!</p>
<p>```json</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"influxdb/influxdb-php"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~1.3"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>A big thanks at all our contributors!</p>
Swarm scales docker for free2015-12-14T10:08:27+00:00http://gianarb.it/blog/swarm-scales-your-containter-for-free<blockquote class="twitter-tweet tw-align-center" data-lang="en"><p lang="en" dir="ltr">An ocean of containers! With docker and swarm.. <a href="https://t.co/1dXoZYS3ZA">https://t.co/1dXoZYS3ZA</a> <a href="https://twitter.com/hashtag/docker?src=hash">#docker</a></p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/696620821931036672">February 8, 2016</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p><a href="https://github.com/gianarb/gourmet">Gourmet</a> is a work in progress application
that allow you to execute little applications on an isolated environment, it
dowloads your manifest and runs it in a container.
I start this application to improve my go knowledge and to work with the Docker API
I am happy to share my idea and my tests with Swam an easy way to scale this type of application.</p>
<p>Gourmet exposes an HTTP API available at the <code class="highlighter-rouge">/project</code> endpoint that accept a JSON request body like:</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"img"</span><span class="p">:</span><span class="w"> </span><span class="s2">"gourmet/php"</span><span class="p">,</span><span class="w">
</span><span class="s2">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://ramdom-your-source.net/gourmet.zip"</span><span class="p">,</span><span class="w">
</span><span class="s2">"env"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="s2">"AWS_KEY=EXAMPLE"</span><span class="p">,</span><span class="w">
</span><span class="s2">"AWS_SECRET="</span><span class="p">,</span><span class="w">
</span><span class="s2">"AWS_QUEUE=https://sqs.eu-west-1.amazonaws.com/test"</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<ul>
<li><code class="highlighter-rouge">img</code> is the started point docker image</li>
<li><code class="highlighter-rouge">source</code> is your script</li>
<li><code class="highlighter-rouge">env</code> is a list of environment variables that you can use on your script</li>
</ul>
<p>During my test I use this <a href="https://github.com/gianarb/gourmet-php-example">php script</a> that send a message on SQS.</p>
<p>Your script has a console entrypoint executables in this path <code class="highlighter-rouge">/bin/console</code> and
gourmet uses it to run your program.</p>
<p>To integrate it with Docker I used <code class="highlighter-rouge">fsouza/go-dockerclient</code> an open source
library written in go.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">container</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Docker</span><span class="o">.</span><span class="n">CreateContainer</span><span class="p">(</span><span class="n">docker</span><span class="o">.</span><span class="n">CreateContainerOptions</span><span class="p">{</span><span class="x">
</span><span class="s">""</span><span class="p">,</span><span class="x">
</span><span class="o">&amp;</span><span class="n">docker</span><span class="o">.</span><span class="n">Config</span><span class="p">{</span><span class="x">
</span><span class="n">Image</span><span class="o">:</span><span class="x"> </span><span class="n">img</span><span class="p">,</span><span class="x">
</span><span class="n">Cmd</span><span class="o">:</span><span class="x"> </span><span class="p">[]</span><span class="kt">string</span><span class="p">{</span><span class="s">"sleep"</span><span class="p">,</span><span class="x"> </span><span class="s">"1000"</span><span class="p">},</span><span class="x">
</span><span class="n">WorkingDir</span><span class="o">:</span><span class="x"> </span><span class="s">"/tmp"</span><span class="p">,</span><span class="x">
</span><span class="n">AttachStdout</span><span class="o">:</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x">
</span><span class="n">AttachStderr</span><span class="o">:</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x">
</span><span class="n">Env</span><span class="o">:</span><span class="x"> </span><span class="n">envVars</span><span class="p">,</span><span class="x">
</span><span class="p">},</span><span class="x">
</span><span class="no">nil</span><span class="p">,</span><span class="x">
</span><span class="p">})</span></code></pre></figure>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<p>This is a snippet that can be used to create a new container.
With the container started I use the exec feature to
extract your source and to run it.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">exec</span><span class="p">,</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Docker</span><span class="o">.</span><span class="n">CreateExec</span><span class="p">(</span><span class="n">docker</span><span class="o">.</span><span class="n">CreateExecOptions</span><span class="p">{</span><span class="x">
</span><span class="n">Container</span><span class="o">:</span><span class="x"> </span><span class="n">containerId</span><span class="p">,</span><span class="x">
</span><span class="n">AttachStdin</span><span class="o">:</span><span class="x"> </span><span class="no">true</span><span class="p">,</span><span class="x">
</span><span class="n">AttachStdout</span><span class="o">:</span><span class="x"> </span><span class="no">true</span><span class="p">,</span><span class="x">
</span><span class="n">AttachStderr</span><span class="o">:</span><span class="x"> </span><span class="no">true</span><span class="p">,</span><span class="x">
</span><span class="n">Tty</span><span class="o">:</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x">
</span><span class="n">Cmd</span><span class="o">:</span><span class="x"> </span><span class="n">command</span><span class="p">,</span><span class="x">
</span><span class="p">})</span><span class="x">
</span><span class="k">if</span><span class="x"> </span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="p">;</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Docker</span><span class="o">.</span><span class="n">StartExec</span><span class="p">(</span><span class="n">exec</span><span class="o">.</span><span class="n">ID</span><span class="p">,</span><span class="x"> </span><span class="n">docker</span><span class="o">.</span><span class="n">StartExecOptions</span><span class="p">{</span><span class="x">
</span><span class="n">Detach</span><span class="o">:</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x">
</span><span class="n">Tty</span><span class="o">:</span><span class="x"> </span><span class="no">false</span><span class="p">,</span><span class="x">
</span><span class="n">RawTerminal</span><span class="o">:</span><span class="x"> </span><span class="no">true</span><span class="p">,</span><span class="x">
</span><span class="n">OutputStream</span><span class="o">:</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Stream</span><span class="p">,</span><span class="x">
</span><span class="n">ErrorStream</span><span class="o">:</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Stream</span><span class="p">,</span><span class="x">
</span><span class="p">})</span></code></pre></figure>
<p>After each build Gourmet cleans all and destroies the environment.</p>
<figure class="highlight"><pre><code class="language-go" data-lang="go"><span class="n">err</span><span class="x"> </span><span class="o">:=</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Docker</span><span class="o">.</span><span class="n">KillContainer</span><span class="p">(</span><span class="n">docker</span><span class="o">.</span><span class="n">KillContainerOptions</span><span class="p">{</span><span class="n">ID</span><span class="o">:</span><span class="x"> </span><span class="n">containerId</span><span class="p">})</span><span class="x">
</span><span class="n">err</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="n">dr</span><span class="o">.</span><span class="n">Docker</span><span class="o">.</span><span class="n">RemoveContainer</span><span class="p">(</span><span class="n">docker</span><span class="o">.</span><span class="n">RemoveContainerOptions</span><span class="p">{</span><span class="n">ID</span><span class="o">:</span><span class="x"> </span><span class="n">containerId</span><span class="p">,</span><span class="x"> </span><span class="n">RemoveVolumes</span><span class="o">:</span><span class="x"> </span><span class="no">true</span><span class="p">})</span><span class="x">
</span><span class="k">if</span><span class="p">(</span><span class="n">err</span><span class="x"> </span><span class="o">!=</span><span class="x"> </span><span class="no">nil</span><span class="p">)</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="n">err</span><span class="p">;</span><span class="x">
</span><span class="p">}</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="no">nil</span></code></pre></figure>
<p>At the moment it is gourmet, It could be different hypothetical use cases:</p>
<ul>
<li>high separated task</li>
<li>run a testsuite</li>
<li>dispatch specific functions</li>
</ul>
<p>A microservice to work with docker container easily.</p>
<p>I thought about an easy way to scale this application and I found
<a href="https://docs.docker.com/swarm/">Swarm</a>, it is a native cluster for docker and
it seems awesome in first because it is compatibile with the docker api.</p>
<h2 id="swarm">Swarm</h2>
<p>A Docker Swarm’s cluster is very easy to setup, I worked on this project
<a href="https://github.com/gianarb/vagrant-swarm">vagrant-swarm</a> to create a local
environment but <a href="https://docs.docker.com/swarm/install-manual/">the official
documentation</a> is easy to follow.</p>
<p>Swarm’s cluster has two actors:</p>
<ul>
<li>A master is the entrypoint of your requests, it provide an HTTP
api compatible with docker.</li>
<li>A series of nodes that communicate with the master.</li>
</ul>
<p>During this example we will work with 1 master and 2 nodes.
Build this machine with virtualbox , with another tool, or in cloud is not a
problem and <a href="https://docs.docker.com/engine/installation/">install docker</a>.</p>
<p>Into the master pull swarm and create a cluster identifier.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker pull swarm
docker run <span class="nt">--rm</span> swarm create
docker run <span class="nt">--name</span> swarm_master <span class="nt">-d</span> <span class="nt">-p</span> &lt;manager_port&gt;:2375 swarm manage token://&lt;cluster_id&gt;</code></pre></figure>
<p><code class="highlighter-rouge">swarm create</code> returns a cluster_id use them to start the manager and the
<code class="highlighter-rouge">manager_ip</code> is the ip of your master server.</p>
<p>Now go into the node, because we must do few things.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker daemon <span class="nt">-H</span> tcp://0.0.0.0:2375 <span class="nt">-H</span> unix:///var/run/docker.sock
docker run <span class="nt">-d</span> swarm join <span class="nt">--addr</span><span class="o">=</span>&lt;node_ip:2375&gt; token://&lt;cluster_id&gt;</code></pre></figure>
<p>When <code class="highlighter-rouge">cluster_id</code> is the id created in the previous step and the <code class="highlighter-rouge">node_id</code> is the ip
of your current node.
Enter into the master and restart your manager container</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker restart swarm_master</code></pre></figure>
<p>Now we are ready to test if all it’s up.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker <span class="nt">-H</span> tcp://0.0.0.0:2375 info</code></pre></figure>
<p>Replace <code class="highlighter-rouge">0.0.0.0.0</code> with your master ip if you are in the same server.
You’ll wait this type of response</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$.</span> <span class="nb">sudo </span>docker <span class="nt">-H</span> tcp://192.168.13.1:2375 info
Containers: 1
Images: 1
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 2
vagrant-ubuntu-vivid-64: 192.168.13.101:2375
└ Status: Healthy
└ Containers: 1
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 513.5 MiB
└ Labels: <span class="nv">executiondriver</span><span class="o">=</span>native-0.2, <span class="nv">kernelversion</span><span class="o">=</span>3.19.0-43-generic, <span class="nv">operatingsystem</span><span class="o">=</span>Ubuntu 15.04, <span class="nv">storagedriver</span><span class="o">=</span>aufs
vagrant-ubuntu-vivid-64: 192.168.13.102:2375
└ Status: Healthy
└ Containers: 0
└ Reserved CPUs: 0 / 1
└ Reserved Memory: 0 B / 513.5 MiB
└ Labels: <span class="nv">executiondriver</span><span class="o">=</span>native-0.2, <span class="nv">kernelversion</span><span class="o">=</span>3.19.0-43-generic, <span class="nv">operatingsystem</span><span class="o">=</span>Ubuntu 15.04, <span class="nv">storagedriver</span><span class="o">=</span>aufs
CPUs: 1
Total Memory: 513.5 MiB
Name: f5e23167339e</code></pre></figure>
<p>Gourmet is a set of environment variables to create a connection with docker
api, in particular this function
<a href="https://godoc.org/github.com/fsouza/go-dockerclient#NewClientFromEnv">NewClientFromEnv</a>
and the <code class="highlighter-rouge">DOCKER_HOST</code> parameter.</p>
<p>Docker Swarm supports the same Docker API in this way gourmet uses more nodes.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$ DOCKER_HOST</span><span class="o">=</span><span class="s2">"tcp://192.168.13.1:2333"</span> ./gourmet api</code></pre></figure>
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=90cae01e86" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">News from Docker Planet</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_90cae01e86" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
<div class="row">
<div class="col-md-12">
<a href="//gianarb.it/planet/docker.html"><b>Docker Planet</b></a> is a collection of posts, events and other
information about Docker and all the container ecosystem. Read other
articles and stay in touch, subsbrite to the newsletter.
</div>
</div>
Docker and wordpress for a better world2015-12-14T10:08:27+00:00http://gianarb.it/blog/wordpress-docker<blockquote class="twitter-tweet tw-align-center" lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/hashtag/docker?src=hash">#docker</a> and <a href="https://twitter.com/hashtag/wordpress?src=hash">#wordpress</a> for a better world.. <a href="https://t.co/o9c6YXvsl3">https://t.co/o9c6YXvsl3</a> Blogpost after my talk <a href="https://twitter.com/CodemotionIT">@CodemotionIT</a> How and Why? <a href="https://twitter.com/awscloud">@awscloud</a></p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/679241680797700096">December 22, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I am trying to represent a typical wordpress infrastructure</p>
<p><img src="/img/posts/2015-12-16/wp-infra.png" alt="Wordpress typical infrastructure" /></p>
<p><strong>Isolation</strong>: every single wordpress share all with the others, filesystem,
memory, database.</p>
<p>This lack of isolation causes different problems:</p>
<ul>
<li>The monitoring of each installation is harder.</li>
<li>We share security problems</li>
<li>We don’t have the freedom to work without the fear or blocking 100 customers</li>
</ul>
<p>We are overwhelmed by the problems</p>
<p><img src="/img/posts/2015-12-16/problem.png" alt="Problem" /></p>
<h2 id="lxc-container">LXC Container</h2>
<blockquote>
<p>it is an operating-system-level virtualization environment for running multiple
isolated Linux systems (containers) on a single Linux control host.</p>
<p>by wikipedia</p>
</blockquote>
<p>Wikipedia helps me to resolve one problem (theory), container is <strong>isolated
Linux System</strong></p>
<div class="post row">
<div class="col-md-12">
<!--<div class="bs-callout bs-callout-info row">-->
<!--<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" class="img-fluid"></a>-->
<!--</div>-->
</div>
</div>
<h2 id="docker">Docker</h2>
<p>Docker borns as wrap of LXC container but now we use an own implementation
<a href="https://github.com/opencontainers/runc">runc</a> to serve your application ready
to go in an isolate environment, with own filesystem and dependencies.</p>
<p>Worpdress in this implemetation has two containers, one to provide apache and
php and one for mysql database. This is an example of Dockerfile, it describes
how a docker container works it is very simple to understand, from this example
there are different keywords</p>
<ul>
<li><code class="highlighter-rouge">FROM</code> describes the image that we use as start point.</li>
<li><code class="highlighter-rouge">RUN</code> run a command.</li>
<li><code class="highlighter-rouge">EXPOSE</code> describes ports to open during a link, in this case MySql runs on
the default port 3306.</li>
<li><code class="highlighter-rouge">CMD</code> is the default command used during the run console command.</li>
</ul>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">FROM ubuntu
RUN dpkg-divert <span class="nt">--local</span> <span class="nt">--rename</span> <span class="nt">--add</span> /sbin/initctl
RUN ln <span class="nt">-s</span> /bin/true /sbin/initctl
RUN <span class="nb">echo</span> <span class="s2">"deb http://archive.ubuntu.com/ubuntu precise main universe"</span> <span class="o">&gt;</span> /etc/apt/sources.list
RUN apt-get update
RUN apt-get <span class="nt">-y</span> install mysql-server
EXPOSE 3306
CMD <span class="o">[</span><span class="s2">"/usr/bin/mysqld_safe"</span><span class="o">]</span></code></pre></figure>
<p>Very easy to read, it is a list of commands!
We are only write a container definition, now we can build it!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker build <span class="nt">-t</span> gianarb/mysql .</code></pre></figure>
<p>In order to increase the value of this article and to use stable images I will
use the official <a href="https://hub.docker.com/_/mysql/">mysql</a> and
<a href="https://hub.docker.com/_/wordpress/">wordpress</a> images.</p>
<p>Download this images</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker pull wordpress
docker pull mysql</code></pre></figure>
<p>We are ready to run all! Dockerfile is only a way to describe each single
container, and the pull command downloads online container ready to work, it is
a good way to reuse your or other containers.</p>
<p>We downloaded mysql and wordpress, with the run command we start them and we
define our connections</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker run <span class="se">\</span>
<span class="nt">--name</span> mysql <span class="se">\</span>
<span class="nt">-p</span> 3306:3306 <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">MYSQL_ROOT_PASSWORD</span><span class="o">=</span>passwd mysql
docker run <span class="nt">-e</span> <span class="nv">WORDPRESS_DB_HOST</span><span class="o">=</span>wp1.database.prod <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">WORDPRESS_DB_USER</span><span class="o">=</span>root <span class="se">\</span>
<span class="nt">-e</span> <span class="nv">WORDPRESS_DB_PASSWORD</span><span class="o">=</span>help_me <span class="se">\</span>
<span class="nt">-p</span> 8080:80 <span class="se">\</span>
<span class="nt">-d</span> <span class="nt">--name</span> wp1 <span class="se">\</span>
<span class="nt">--link</span> wp.database.prod:mysql wordpress</code></pre></figure>
<p>I can try to explain this commands, it run two containers:</p>
<ul>
<li>The name of the first container is mysql and it uses the <code class="highlighter-rouge">mysql</code> image, we
use -p flag to expose mysql port now you can use phpmyadmin or other client
to fetch the data but remember that is not a good practice.</li>
<li>The second container called wp1 uses the image <code class="highlighter-rouge">gianarb/wordpress</code> forward
the container port 80 (apache) on host 8080, that in this case it is the way
to see the site. –link flag is the correct way to consume mysql outside the
main container, in this particular case we could use wp.database.prod how url
to connect at mysql from our worpdress container, awesome!</li>
<li>Docker image supports environment variable <code class="highlighter-rouge">ENV</code> for example we can use them
to configure our services, in this case to set root password in mysql and to
configure worpdress’s database connection</li>
</ul>
<p>We are ready! Now you have a worpdress ready to go on port 8080.</p>
<h2 id="docker-compose">Docker Compose</h2>
<p>To save time and to increase reusability we can use
<a href="https://docs.docker.com/compose/">docker-compose</a> tool
that helps us to manage multi-container infrastructures, in this case one for
mysql and one for wordpress.
In practice we can describe all work did above in a <code class="highlighter-rouge">docker-compose.yml</code> file:</p>
<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">wp</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">wordpress</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">8081:80</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">WORDPRESS_DB_HOST</span><span class="pi">:</span> <span class="s">wp1.database.prod</span>
<span class="na">WORDPRESS_DB_USER</span><span class="pi">:</span> <span class="s">root</span>
<span class="na">WORDPRESS_DB_PASSWORD</span><span class="pi">:</span> <span class="s">help_me</span>
<span class="na">links</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">wp1.database.prod:mysql</span>
<span class="na">mysql</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">mysql:5.7</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="na">MYSQL_ROOT_PASSWORD</span><span class="pi">:</span> <span class="s">help_me</span></code></pre></figure>
<p>Now we can run</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">docker-compose build <span class="nb">.</span>
docker-compose up</code></pre></figure>
<p>To prepare and start our infrastructure. Now we have one wordpress with own
mysql that run on port 8081. We can change wordpress port to start new isolate
wordpress installation.</p>
<p class="text-center">
<iframe src="//giphy.com/embed/l41lYCDgxP6OFBruE" width="480" height="268" frameborder="0" class="giphy-embed" allowfullscreen=""></iframe><p><a href="http://giphy.com/gifs/foxtv-win-ricky-gervais-emmys-2015-l41lYCDgxP6OFBruE">via
GIPHY</a></p>
</p>
<h2 id="in-cloud-with-aws-ecs">In Cloud with AWS ECS</h2>
<p>We won a battle but the war is too long, we can not use our PC as server. In
this article I propose <a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html">AWS Elastic Container
Service</a>
a new AWS service that helps us to manage containers, why this service? Because
it is Docker and Docker Composer like, it’s managed by AWS, maybe there are
more flexible solutions, Swarm, Kubernetes but it is a good start point.</p>
<p><img src="/img/posts/2015-12-16/ecs.png" alt="AWS Elastic Container Service" /></p>
<p>A services of keywords to understand how it works:</p>
<ul>
<li><strong>Container instance</strong>: An Amazon EC2 that is running the Amazon ECS Agent. It has been registered into the ECS.</li>
<li><strong>Cluster</strong>: It is a pool of Container instances</li>
<li><strong>Task definition</strong>: A description of an application that contains one or more container definitions</li>
<li>Each Task definition running is a <strong>Task</strong></li>
</ul>
<h3 id="in-practice">In practice</h3>
<ol>
<li>Create a cluster</li>
</ol>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ecs-cli configure <span class="se">\</span>
<span class="nt">--region</span> eu-west-1 <span class="se">\</span>
<span class="nt">--cluster</span> wps <span class="se">\</span>
<span class="nt">--access-key</span> apikey <span class="se">\</span>
<span class="nt">--secret-key</span> secreyKey</code></pre></figure>
<ol>
<li>Up nodes (one in this case)</li>
</ol>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ecs-cli up <span class="nt">--keypair</span> key-ecs <span class="se">\</span>
<span class="nt">--capability-iam</span> <span class="se">\</span>
<span class="nt">--size</span> 1 <span class="se">\</span>
<span class="nt">--instance-type</span> t2.medium</code></pre></figure>
<ol>
<li>Push your first task!</li>
</ol>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ecs-cli compose <span class="nt">--file</span> docker-compose.yml <span class="se">\</span>
<span class="nt">--project-name</span> wp1 up</code></pre></figure>
<ol>
<li>Follow the status of your tasks</li>
</ol>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ecs-cli ps</code></pre></figure>
<p>You can use another docker-compose.yml with a different wordpress port to build
another task with another worpdress!</p>
<h2 id="now-is-only-a-problem-of-url">Now is only a problem of URL</h2>
<p>We are different isolated worpdress online, but they are an ip and different
ports, maybe our customers would use a domain name for example.
I don’t know if this solution is ready to run in production and it is good to
run more and more wordpress but a good service to turn and proxy requests is
HaProxy. This is an example of configuration for our use case:</p>
<p>wp1.gianarb.it and wp1.gianarb.it are two our customers and 54.229.190.73:8080,
54.229.190.73:8081 are our wordpress.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">...
frontend wp_mananger
<span class="nb">bind</span> :80
acl host_wp1 hdr<span class="o">(</span>host<span class="o">)</span> <span class="nt">-i</span> wp1.gianarb.it
acl host_wp2 hdr<span class="o">(</span>host<span class="o">)</span> <span class="nt">-i</span> wp2.gianarb.it
use_backend backend_wp1 <span class="k">if </span>host_wp1
use_backend backend_wp2 <span class="k">if </span>host_wp2
backend backend_wp1
server server1 54.229.190.73:8080 check
backend backend_wp2
server server2 54.229.190.73:8081 check</code></pre></figure>
<p>Note: This configuration increase the scalability of our system, because we can
add other service in order to support more traffic.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">backend backend_wp1
server server1 54.229.190.73:8080 check
server server1 54.229.190.12:8085 check
server server1 54.229.190.15:80 check</code></pre></figure>
<h3 id="there-are-other-solutions">There are other solutions</h3>
<ul>
<li>Nginx</li>
<li>Consul to increase the stability and the scalability of our endpoint</li>
</ul>
<div class="alert alert-info" role="alert">
This article is based on my presentation at <a href="http://gianarb.it/codemotion-2015/" target="_blank">Codemotion 2015</a>
</div>
<div class="alert alert-success" role="alert">
Thanks for review <a href="https://twitter.com/fntlnz" target="_blank">Lorenzo</a>! I'm in Ireland from 3 weeks but I am not ready to
write an article without your english review!
</div>
<!-- Begin MailChimp Signup Form -->
<link href="//cdn-images.mailchimp.com/embedcode/horizontal-slim-10_7.css" rel="stylesheet" type="text/css" />
<style type="text/css">
#mc_embed_signup{background:#fff; clear:left; font:14px Helvetica,Arial,sans-serif; width:100%;}
/* Add your own MailChimp form style overrides in your site stylesheet or in this style block.
We recommend moving this block and the preceding CSS link to the HEAD of your HTML file. */
</style>
<div id="mc_embed_signup">
<form action="//gianarb.us2.list-manage.com/subscribe/post?u=03581a38fd820a06b0898c802&amp;id=90cae01e86" method="post" id="mc-embedded-subscribe-form" name="mc-embedded-subscribe-form" class="validate" target="_blank" novalidate="">
<div id="mc_embed_signup_scroll">
<label for="mce-EMAIL">News from Docker Planet</label>
<input type="email" value="" name="EMAIL" class="email" id="mce-EMAIL" placeholder="email address" required="" />
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups-->
<div style="position: absolute; left: -5000px;" aria-hidden="true"><input type="text" name="b_03581a38fd820a06b0898c802_90cae01e86" tabindex="-1" value="" /></div>
<div class="clear"><input type="submit" value="Subscribe" name="subscribe" id="mc-embedded-subscribe" class="button" /></div>
</div>
</form>
</div>
<!--End mc_embed_signup-->
<div class="row">
<div class="col-md-12">
<a href="//gianarb.it/planet/docker.html"><b>Docker Planet</b></a> is a collection of posts, events and other
information about Docker and all the container ecosystem. Read other
articles and stay in touch, subsbrite to the newsletter.
</div>
</div>
FastEventManager, only an event manager2015-11-01T00:00:00+00:00http://gianarb.it/blog/fast-event-manager-only-an-event-manager<blockquote>
<p>The Event-Driven Messaging is a design pattern, applied within the
service-orientation design paradigm in order to enable the service consumers,
which are interested in events that occur within the periphery of a service
provider, to get notifications about these events as and when they occur
without resorting to the traditional inefficientpolling based mechanism.
by. <a href="https://en.wikipedia.org/wiki/Event-Driven_Messaging">wiki</a></p>
</blockquote>
<p>In PHP there are different implementation of this pattern, but <a href="https://github.com/gianarb/fast-event-manager">I tried to write
my idea</a>.
An easy to understand and to extends event manager based on regex.</p>
<p>Why? Because it is a good way to match strings, it is flexible and powerful.
As it is smart and little and it can be used as basis for custom implementation.
it resolves a regex and triggers events It supports a priority to order
triggered listeners.</p>
<h2 id="install">Install</h2>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">composer require gianarb/fast-event-manager</code></pre></figure>
<h2 id="getting-started">Getting Started</h2>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">require</span> <span class="nx">__DIR__</span><span class="o">.</span><span class="s2">"/vendor/autoload.php"</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">GianArb\FastEventManager</span><span class="p">;</span>
<span class="nv">$eventManager</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">FastEventManager</span><span class="p">();</span>
<span class="nv">$eventManager</span><span class="o">-&gt;</span><span class="na">attach</span><span class="p">(</span><span class="s2">"user_saved"</span><span class="p">,</span> <span class="k">function</span><span class="p">(</span><span class="nv">$event</span><span class="p">)</span> <span class="p">{</span>
<span class="p">});</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Entity\User</span><span class="p">();</span>
<span class="nv">$eventManager</span><span class="o">-&gt;</span><span class="na">trigger</span><span class="p">(</span><span class="s2">"/user_saved/"</span><span class="p">,</span> <span class="nv">$event</span><span class="p">);</span></code></pre></figure>
<p>Each listener has a priority (default = 0), it describe the order of execution</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="nv">$eventManager</span><span class="o">-&gt;</span><span class="na">attach</span><span class="p">(</span><span class="s2">"wellcome"</span><span class="p">,</span> <span class="k">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">echo</span> <span class="s2">" dev!"</span><span class="p">;</span>
<span class="p">},</span> <span class="mi">100</span><span class="p">);</span>
<span class="nv">$eventManager</span><span class="o">-&gt;</span><span class="na">attach</span><span class="p">(</span><span class="s2">"wellcome"</span><span class="p">,</span> <span class="k">function</span><span class="p">()</span> <span class="p">{</span>
<span class="k">echo</span> <span class="s2">"Hello"</span><span class="p">;</span>
<span class="p">},</span> <span class="mi">345</span><span class="p">);</span>
<span class="nv">$eventManager</span><span class="o">-&gt;</span><span class="na">trigger</span><span class="p">(</span><span class="s2">"/wellcome/"</span><span class="p">);</span>
<span class="o">//</span><span class="nx">output</span> <span class="s2">"Hello dev!"</span></code></pre></figure>
<p>I wrote this library because there are a lot of solutions that implement this
pattern but they are verbose, this is only an event manager if you search other
features you can extends it or you can use differents implementations.
On top of this library you can write your library to build an event manager ready
to use with your team in your applications.</p>
<p>This is a good solution because it is easy, ~31 line of code to trigger events
without fear to inherit many line of codes and unused features to maintain.</p>
Penny PHP framework made of components2015-10-27T23:08:27+00:00http://gianarb.it/blog/penny-framework-made-of-components<blockquote class="twitter-tweet tw-align-center" lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/hashtag/pennyphp?src=hash">#pennyphp</a> <a href="https://t.co/tsA2nE09GM">https://t.co/tsA2nE09GM</a> Why and what?! o.O <a href="https://twitter.com/hashtag/php?src=hash">#php</a> <a href="https://twitter.com/hashtag/framework?src=hash">#framework</a> to build <a href="https://twitter.com/hashtag/microservices?src=hash">#microservices</a> and application &quot;consciously&quot;</p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/659762064446083073">October 29, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p class="text-center">
<iframe src="https://ghbtns.com/github-btn.html?user=pennyphp&amp;repo=penny&amp;type=star&amp;count=true&amp;size=large" frameborder="0" scrolling="0" width="160px" height="30px"></iframe>
</p>
<p>The PHP ecosystem is mature, there are a lot of libraries that help you to write
good and custom applications. Too much libraries require a strong knowledge to
avoid the problem of maintainability and they also open a world made on specific
implementations for specific use cases.</p>
<p>A big framework adds a big overhead under your business logic sometimes, and some
of those unused features could cause maintainability problems and chaos.</p>
<p>Spending too much time reading the docs could be a problem, do you think you are
a system integrator and not a developer?! These are different works!</p>
<p>We are writing <a href="https://github.com/pennyphp/penny">penny</a> to share this idea.
This is a middleware, event driven framework to build the perfect
implementation for your specific project. The starting point we chose is made of:</p>
<ul>
<li><a href="https://github.com/zendframework/zend-diactoros">Zend\Diactoros</a> PSR-7 HTTP
library</li>
<li><a href="https://github.com/zendframework/zend-eventmanager">Zend\EventManager</a> to
design the application flow</li>
<li><a href="https://php-di">PHP-DI</a> DiC library</li>
<li><a href="https://github.com/nikic/FastRoute">FastRouter</a> because it is fast and easy to
use</li>
</ul>
<p>but we are working to replace every part of penny with the libraries perfect
for your use case.</p>
<p>Are you curious to try this idea? We are writing a big documentation around penny.
<a href="http://docs.pennyphp.org/en/latest/">docs.pennyphp.org/en/latest</a></p>
<p>And we have a set of use cases:</p>
<ul>
<li><a href="https://github.com/pennyphp/penny-classic-app">pennyphp/penny-classic-app</a>
builds with plates</li>
<li><a href="https://github.com/pennyphp/bookshelf">pennyphp/bookshelf</a> builds with
doctrine, twig</li>
<li><a href="https://github.com/gianarb/twitter-uservice">gianarb/twitter-uservice</a> gets
the last tweet from <code class="highlighter-rouge">#AngularConf15</code> hashtag</li>
</ul>
<p><a href="https://github.com/pennyphp/penny/issues?utf8=%E2%9C%93&amp;q=is%3Aissue">Share your experience!</a></p>
vim composer 0.3.0 is ready2015-09-15T23:08:27+00:00http://gianarb.it/blog/php-and-vim-composer-release-0-3-0<blockquote align="center" class="twitter-tweet" data-cards="hidden" lang="en"><p lang="en" dir="ltr"><a href="https://twitter.com/hashtag/vimForPHP?src=hash">#vimForPHP</a> <a href="https://t.co/EdczdpCrRc">https://t.co/EdczdpCrRc</a> <a href="https://twitter.com/hashtag/php?src=hash">#php</a> <a href="https://twitter.com/hashtag/vim?src=hash">#vim</a> Release 0.3.0 <a href="https://twitter.com/hashtag/composer?src=hash">#composer</a> plugin is ready! Thanks <a href="https://twitter.com/sensorario">@sensorario</a> for your work!</p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/641674841192574976">September 9, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>I’m very happy to announce release 0.3.0 of <a href="https://github.com/vim-php/vim-composer">vim-composer</a>.
This plugin builds a good integration between VIM and <a href="https://getcomposer.org">composer</a> the strong dependency manager for PHP.</p>
<h2 id="changelog">Changelog</h2>
<ul>
<li><a href="https://github.com/vim-php/vim-composer/pull/18">#18</a> Added missing ComposerUpdate function</li>
<li><a href="https://github.com/vim-php/vim-composer/pull/21">#21</a> Added missing CONTRIBUTING.md file</li>
<li><a href="https://github.com/vim-php/vim-composer/pull/20">#20</a> Require and/or init commands</li>
</ul>
<p>Now this plugin serve new function to require specific package. Update it and map new function <code class="highlighter-rouge">:ComposerRequireFunc</code>.</p>
Staging environment on demand with AWS Cloudformation2015-07-08T09:08:27+00:00http://gianarb.it/blog/stagin-environment-on-demand-with-aws-cloudformation<blockquote class="twitter-tweet tw-align-center" lang="en"><p lang="en" dir="ltr">Staging environment on demand. To Work on <a href="https://twitter.com/hashtag/AWS?src=hash">#AWS</a> low level with <a href="https://twitter.com/hashtag/cloudformation?src=hash">#cloudformation</a> <a href="http://t.co/VWBR129637">http://t.co/VWBR129637</a> <a href="https://twitter.com/hashtag/cloud?src=hash">#cloud</a> <a href="https://twitter.com/hashtag/devops?src=hash">#devops</a></p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/621691855810494464">July 16, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<h2 id="staging-environment">Staging Environment</h2>
<p>There are few environments during my developer workflow, today I chose a little example:</p>
<ul>
<li>Production enviroment always exists, it runs the stable application and you can not use it for your test.</li>
<li><strong>Staging</strong> enviroment is a “pre-production” state.</li>
<li>Develop enviroment is instable and it runs new features and fixes, here there’s the work of all team but it’s not ready to go in production.</li>
</ul>
<p><img src="/img/cloudformation-staging/staging.jpg" alt="Staging graph" /></p>
<p>Staging environment in my opinion could be “volatile” version, we use it when our product is ready to go in production for the last time it was unused. Maybe this statement isn’t real in your work but if you think a little team of consultants that
work on different projects maybe this words have a sense.</p>
<h2 id="aws-cloudformation">AWS Cloudformation</h2>
<p>CloudFormation is an AWS service that helps you to orchestate all AWS services, you can write a template in JSON and you can use it to create an infrastructure with one click.
This solution helps me to build and destroy this environment and we can pay it only if it’s necessary, if you use a <code class="highlighter-rouge">stagin env == production env</code> it can be very very expensive.
This solution could help you to down cost.</p>
<h2 id="current-infrastructure">Current infrastructure</h2>
<p><img src="/img/cloudformation-staging/infra.jpg" alt="RDS and EC2 infrastructure" /></p>
<p>This is my template to build a simple application Frontend + MySQL (RDS).
In this implementation I build network configuration and I create one instance of RDS and one EC2 (my frontend).
<code class="highlighter-rouge">Parameters</code> key is the list of external parameters that I can use to configure my template, for example database and EC2 key pair, my root’s password..
<code class="highlighter-rouge">Resources</code> key contains description of all actors of this infrastructure.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"Parameters"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"VPCName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"staging"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"VPC name"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"ProjectName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"app"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Project name"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebKey"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"web-key"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Ssh key to log into the web instances"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebInstanceType"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"m3.medium"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Web instance type"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebInstanceImage"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"ami-47a23a30"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Web instance image"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseInstanceType"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"db.m3.medium"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Database instance type"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"mydb"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Database instance's name"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseMasterUsername"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"gianarb"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Name of master user"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseEngineVersion"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"5.6"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"MySQL version"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseUserPassword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"test1234"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"User password"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabasePublicAccess"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseMultiAZ"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Resources"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Staging"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::VPC"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"CidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10.15.0.0/16"</span><span class="p">,</span><span class="w">
</span><span class="s2">"EnableDnsSupport"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"EnableDnsHostnames"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"InstanceTenancy"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VPCName"</span><span class="p">}}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseSubnet1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::Subnet"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"AvailabilityZone"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"eu-west-1a"</span><span class="p">,</span><span class="w">
</span><span class="s2">"CidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10.15.1.0/28"</span><span class="p">,</span><span class="w">
</span><span class="s2">"MapPublicIpOnLaunch"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"db-1a"</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseSubnet2"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::Subnet"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"AvailabilityZone"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"eu-west-1b"</span><span class="p">,</span><span class="w">
</span><span class="s2">"CidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10.15.1.16/28"</span><span class="p">,</span><span class="w">
</span><span class="s2">"MapPublicIpOnLaunch"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"db-1b"</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebSubnet1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::Subnet"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"AvailabilityZone"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"eu-west-1a"</span><span class="p">,</span><span class="w">
</span><span class="s2">"CidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10.15.0.8/28"</span><span class="p">,</span><span class="w">
</span><span class="s2">"MapPublicIpOnLaunch"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"web-1a"</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"RDSSubnet"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::RDS::DBSubnetGroup"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"DBSubnetGroupDescription"</span><span class="p">:</span><span class="w"> </span><span class="s2">"db-prod-subnet-group"</span><span class="p">,</span><span class="w">
</span><span class="s2">"SubnetIds"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseSubnet1"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseSubnet2"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Database"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::RDS::DBInstance"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"AllocatedStorage"</span><span class="p">:</span><span class="w"> </span><span class="s2">"5"</span><span class="p">,</span><span class="w">
</span><span class="s2">"AllowMajorVersionUpgrade"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">false</span><span class="p">,</span><span class="w">
</span><span class="s2">"DBInstanceClass"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"DatabaseInstanceType"</span><span class="p">},</span><span class="w">
</span><span class="s2">"DBName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"DatabaseName"</span><span class="p">},</span><span class="w">
</span><span class="s2">"DBInstanceIdentifier"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"DatabaseName"</span><span class="p">},</span><span class="w">
</span><span class="s2">"Engine"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"MySQL"</span><span class="p">,</span><span class="w">
</span><span class="s2">"EngineVersion"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"DatabaseEngineVersion"</span><span class="p">},</span><span class="w">
</span><span class="s2">"DBSubnetGroupName"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"RDSSubnet"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"MasterUsername"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseMasterUsername"</span><span class="p">},</span><span class="w">
</span><span class="s2">"MasterUserPassword"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseUserPassword"</span><span class="p">},</span><span class="w">
</span><span class="s2">"MultiAZ"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"VPCSecurityGroups"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseSG"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"PubliclyAccessible"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabasePublicAccess"</span><span class="p">},</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Fn::Join"</span><span class="p">:[</span><span class="s2">"."</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="s2">"db"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ProjectName"</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"VPCName"</span><span class="p">}]]}</span><span class="w"> </span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebInstance"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::Instance"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"ImageId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WebInstanceImage"</span><span class="p">},</span><span class="w">
</span><span class="s2">"InstanceType"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WebInstanceType"</span><span class="p">},</span><span class="w">
</span><span class="s2">"KeyName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WebKey"</span><span class="p">},</span><span class="w">
</span><span class="s2">"BlockDeviceMappings"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"DeviceName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"/dev/sdm"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Ebs"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"VolumeType"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"io1"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Iops"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"200"</span><span class="p">,</span><span class="w">
</span><span class="s2">"DeleteOnTermination"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"false"</span><span class="p">,</span><span class="w">
</span><span class="s2">"VolumeSize"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"20"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"DeviceName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"/dev/sdk"</span><span class="p">,</span><span class="w">
</span><span class="s2">"NoDevice"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"SubnetId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="s2">"Ref"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"WebSubnet1"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="s2">"SecurityGroupIds"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WebSG"</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"StagingZone"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::Route53::HostedZone"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Fn::Join"</span><span class="p">:[</span><span class="s2">"."</span><span class="p">,</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ProjectName"</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"VPCName"</span><span class="p">}]]},</span><span class="w">
</span><span class="s2">"VPCs"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"VPCId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="p">},</span><span class="w"> </span><span class="s2">"VPCRegion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"eu-west-1"</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"StagingInternetGateway"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::InternetGateway"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="s2">"Key"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Fn::Join"</span><span class="p">:[</span><span class="s2">"-"</span><span class="p">,</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"VPCName"</span><span class="p">},</span><span class="w"> </span><span class="s2">"igw"</span><span class="p">]]}}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"StagingIgwAttach"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::VPCGatewayAttachment"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"InternetGatewayId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingInternetGateway"</span><span class="p">},</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"StagingRouteTable"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::RouteTable"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"LocalRoute"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::Route"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"DestinationCidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.0.0/0"</span><span class="p">,</span><span class="w">
</span><span class="s2">"GatewayId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingInternetGateway"</span><span class="p">},</span><span class="w">
</span><span class="s2">"RouteTableId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingRouteTable"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Web1LocalRoute"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::SubnetRouteTableAssociation"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"RouteTableId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingRouteTable"</span><span class="p">},</span><span class="w">
</span><span class="s2">"SubnetId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"WebSubnet1"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Db1LocalRoute"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::SubnetRouteTableAssociation"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"RouteTableId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingRouteTable"</span><span class="p">},</span><span class="w">
</span><span class="s2">"SubnetId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseSubnet1"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Db2LocalRoute"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::SubnetRouteTableAssociation"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"RouteTableId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingRouteTable"</span><span class="p">},</span><span class="w">
</span><span class="s2">"SubnetId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"DatabaseSubnet2"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseSG"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::SecurityGroup"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"GroupDescription"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Database security groups"</span><span class="p">,</span><span class="w">
</span><span class="s2">"SecurityGroupIngress"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"IpProtocol"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"tcp"</span><span class="p">,</span><span class="w">
</span><span class="s2">"FromPort"</span><span class="p">:</span><span class="w"> </span><span class="mi">3306</span><span class="p">,</span><span class="w">
</span><span class="s2">"ToPort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"3306"</span><span class="p">,</span><span class="w">
</span><span class="s2">"SourceSecurityGroupId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"WebSG"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"db-sg"</span><span class="p">}],</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebSG"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::SecurityGroup"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"GroupDescription"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Web security groups"</span><span class="p">,</span><span class="w">
</span><span class="s2">"SecurityGroupIngress"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"IpProtocol"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"tcp"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ToPort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">80</span><span class="p">,</span><span class="w">
</span><span class="s2">"FromPort"</span><span class="p">:</span><span class="w"> </span><span class="mi">80</span><span class="p">,</span><span class="w">
</span><span class="s2">"CidrIp"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.0.0/0"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="s2">"IpProtocol"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"tcp"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ToPort"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="mi">22</span><span class="p">,</span><span class="w">
</span><span class="s2">"FromPort"</span><span class="p">:</span><span class="w"> </span><span class="mi">22</span><span class="p">,</span><span class="w">
</span><span class="s2">"CidrIp"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.0.0/0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">],</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"web-sg"</span><span class="p">}],</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseRecordSet"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::Route53::RecordSet"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"HostedZoneId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingZone"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Comment"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"DNS name for database"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Fn::Join"</span><span class="p">:[</span><span class="s2">"."</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="s2">"db"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ProjectName"</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"VPCName"</span><span class="p">}]]},</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"CNAME"</span><span class="p">,</span><span class="w">
</span><span class="s2">"TTL"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"300"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ResourceRecords"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"Fn::GetAtt"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Database"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Endpoint.Address"</span><span class="p">]}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<h2 id="conclusion">Conclusion</h2>
<p>You can load this teamplate in your account and after environment creations you are ready to work with one EC2 instance and one RDS with MySQL 5.6 installed.
You can log into the web interface with key-pair chosen during the creation flow (default ga-eu) and I set default this mysql credential:</p>
<ul>
<li>user gianarb</li>
<li>password test1234</li>
</ul>
<p>But you can change it before running this template because they are <code class="highlighter-rouge">Parameters</code>.
This approach in my opinion is very powerful because you can start versioning your infrastructure and you can delete and restore it quickly because if you delete the cloudformation stack it rollbacks all resources, it is very easy!</p>
<h2 id="trick">Trick</h2>
<p>Parameters node create a form into the AWS CloudFormation console to choose a lot of different variable values, for example name of intances or key-pair to log in your EC2.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"Parameters"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"VPCName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"staging"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"VPC name"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"ProjectName"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"app"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Project name"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"WebKey"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"String"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Default"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"web-key"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Description"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Ssh key to log into the web instances"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<hr class="style-two" />
<p>Resources node contains all elements of your infrastructure, EC2, RDS, VCP.. You can use the parameteters with a simple <code class="highlighter-rouge">Ref Key</code>.
es. <code class="highlighter-rouge">[{"Key": "Name", "Value": "ProjectName"}]</code> describe the name of the specific project into the parameter form.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"Resources"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Staging"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::VPC"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"CidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10.15.0.0/16"</span><span class="p">,</span><span class="w">
</span><span class="s2">"EnableDnsSupport"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"EnableDnsHostnames"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"InstanceTenancy"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"default"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Tags"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VPCName"</span><span class="p">}}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"DatabaseSubnet1"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::EC2::Subnet"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"AvailabilityZone"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"eu-west-1a"</span><span class="p">,</span><span class="w">
</span><span class="s2">"CidrBlock"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"10.15.1.0/28"</span><span class="p">,</span><span class="w">
</span><span class="s2">"MapPublicIpOnLaunch"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="s2">"VpcId"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"Staging"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Tags"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="s2">"Key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Name"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"db-1a"</span><span class="p">}]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<hr class="style-two" />
<p>In your template you can describe VPC and create its subnet. You can also describe the specific resource and you can use it to build another</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="err">WebSubnet</span><span class="mi">1</span><span class="s2">": {
"</span><span class="err">Type</span><span class="s2">" : "</span><span class="err">AWS</span><span class="p">::</span><span class="err">EC</span><span class="mi">2</span><span class="p">::</span><span class="err">Subnet</span><span class="s2">",
"</span><span class="err">Properties</span><span class="s2">" : {
"</span><span class="err">AvailabilityZone</span><span class="s2">" : "</span><span class="err">eu-west</span><span class="mi">-1</span><span class="err">a</span><span class="s2">",
"</span><span class="err">CidrBlock</span><span class="s2">" : "</span><span class="mf">10.15</span><span class="err">.</span><span class="mf">0.8</span><span class="err">/</span><span class="mi">28</span><span class="s2">",
"</span><span class="err">MapPublicIpOnLaunch</span><span class="s2">" : true,
"</span><span class="err">VpcId</span><span class="s2">": {
"</span><span class="err">Ref</span><span class="s2">" : "</span><span class="err">Staging</span><span class="s2">"
},
"</span><span class="err">Tags</span><span class="s2">" : [{"</span><span class="err">Key</span><span class="s2">": "</span><span class="err">Name</span><span class="s2">", "</span><span class="err">Value</span><span class="s2">": "</span><span class="err">web</span><span class="mi">-1</span><span class="err">a</span><span class="s2">"}]
}
},</span></code></pre></figure>
<p>In this example I resumed <code class="highlighter-rouge">Staging</code> VPC to build its subnet.</p>
<hr class="style-two" />
<p>This chapter is insteresting because it creates a RecordSet to map a CNAME DNS in your VPC and now in your Web instances you can resolve MYSql host with <code class="highlighter-rouge">db.app.staging</code>.</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="s2">"DatabaseRecordSet"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"AWS::Route53::RecordSet"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Properties"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"HostedZoneId"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"StagingZone"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"Comment"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"DNS name for database"</span><span class="p">,</span><span class="w">
</span><span class="s2">"Name"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="s2">"Fn::Join"</span><span class="p">:[</span><span class="s2">"."</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="s2">"db"</span><span class="p">,</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ProjectName"</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="s2">"Ref"</span><span class="p">:</span><span class="s2">"VPCName"</span><span class="p">}]]},</span><span class="w">
</span><span class="s2">"Type"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"CNAME"</span><span class="p">,</span><span class="w">
</span><span class="s2">"TTL"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="s2">"300"</span><span class="p">,</span><span class="w">
</span><span class="s2">"ResourceRecords"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="s2">"Fn::GetAtt"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"Database"</span><span class="p">,</span><span class="w"> </span><span class="s2">"Endpoint.Address"</span><span class="p">]}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p><br />
<br />
<br /></p>
<div class="well"><a target="_blank" href="https://twitter.com/EmanueleMinotto">@EmanualeMinotto</a> thanks for trying to fix my bad English</div>
Build your Zend Framework Console Application2015-05-21T23:08:27+00:00http://gianarb.it/blog/zendframework-console-app<blockquote class="twitter-tweet tw-align-center" lang="en"><p lang="en" dir="ltr">Blogpost about console-skeleton-app for your console application <a href="http://t.co/WuVq0GZlxE">http://t.co/WuVq0GZlxE</a> <a href="https://twitter.com/hashtag/PHP?src=hash">#PHP</a> <a href="https://twitter.com/hashtag/ZF?src=hash">#ZF</a> <a href="https://twitter.com/hashtag/console?src=hash">#console</a> <a href="https://twitter.com/hashtag/develop?src=hash">#develop</a></p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/613292048708468736">June 23, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<div class="alert alert-success" role="alert"><strong>Github: </strong>Article written about <a target="_blank" href="https://github.com/gianarb/console-skeleton-app">console-skeleton-app</a> 1.0.0</div>
<p>I’m writing a skeleton app to build console/bash application in PHP.
This project is very easy and it depends on ZF\Console a zfcampus project and Zend\Console builds by ZF community.
I have a todo list for the future but for the time being it’s just a blog post about these two modules.</p>
<ul>
<li>Integration with container system to manage dependency injection</li>
<li>Docs to test your command</li>
<li>Use cases and different implementations</li>
</ul>
<h2 id="zfconsole-and-other-components">ZF\Console and other components</h2>
<ul>
<li><a href="https://github.com/zfcampus/zf-console">ZF\Console</a> is maintained by zfcampus and it is used by Apigility</li>
<li><a href="https://github.com/zendframework/zend-console">zendframework\zend-console</a> is maintained by zendframework, all the info are in the <a href="http://framework.zend.com/manual/current/en/modules/zend.console.introduction.html">documantation</a></li>
</ul>
<h2 id="tree">Tree</h2>
<p>This is my folders structure proposal, there are three entrypoint in the <code class="highlighter-rouge">bin</code> directory, one for bash, one for php and a bat for Window.
I use composer to manage my dependencies and I included .lock file because this project is an APPLICATION not a library..
<code class="highlighter-rouge">/config</code> directory contains only routing definitions but in the future we can add services and other configurations.
<code class="highlighter-rouge">src/Command/</code> contains my commands.</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">├── bin
│ └── console.php
├── composer.json
├── composer.lock
├── config
│ └── routes.php
├── src
│ └── Command
│ ├── Conf.php
│ ├── Database.php
│ └── Download.php
└── vendor
└── ...</code></pre></figure>
<h2 id="bootstrap">Bootstrap</h2>
<p>The Application’s entrypoints are just example and they require few changes.
First we have to change the version in the parameters.php configuration file and also change the application name <code class="highlighter-rouge">'app'</code> to what fits.
To load configurations from different sources I will use the well known <code class="highlighter-rouge">Zend\Config</code> component.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">require</span> <span class="nx">__DIR__</span><span class="o">.</span><span class="s1">'/../vendor/autoload.php'</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Zend\Console\Console</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">ZF\Console\Application</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">ZF\Console\Dispatcher</span><span class="p">;</span>
<span class="nv">$version</span> <span class="o">=</span> <span class="s1">'0.0.1'</span><span class="p">;</span>
<span class="nv">$application</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Application</span><span class="p">(</span>
<span class="s1">'app'</span><span class="p">,</span>
<span class="nv">$version</span><span class="p">,</span>
<span class="k">include</span> <span class="nx">__DIR__</span> <span class="o">.</span> <span class="s1">'/../config/routes.php'</span><span class="p">,</span>
<span class="nx">Console</span><span class="o">::</span><span class="na">getInstance</span><span class="p">(),</span>
<span class="k">new</span> <span class="nx">Dispatcher</span><span class="p">()</span>
<span class="p">);</span>
<span class="nv">$exit</span> <span class="o">=</span> <span class="nv">$application</span><span class="o">-&gt;</span><span class="na">run</span><span class="p">();</span>
<span class="k">exit</span><span class="p">(</span><span class="nv">$exit</span><span class="p">);</span></code></pre></figure>
<h2 id="routes">Routes</h2>
<p><code class="highlighter-rouge">config/routes.php</code> contains router configurations. This is just an example but you can see all options <a href="https://github.com/zfcampus/zf-console#defining-console-routes">here</a>.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">return</span> <span class="p">[</span>
<span class="p">[</span>
<span class="s1">'name'</span> <span class="o">=&gt;</span> <span class="s1">'hello'</span><span class="p">,</span>
<span class="s1">'route'</span> <span class="o">=&gt;</span> <span class="s2">"--name="</span><span class="p">,</span>
<span class="s1">'short_description'</span> <span class="o">=&gt;</span> <span class="s2">"Good morning!! This is a beautiful day"</span><span class="p">,</span>
<span class="s2">"handler"</span> <span class="o">=&gt;</span> <span class="p">[</span><span class="s1">'App\Command\Hello'</span><span class="p">,</span> <span class="s1">'run'</span><span class="p">],</span>
<span class="p">],</span>
<span class="p">];</span></code></pre></figure>
<h2 id="command">Command</h2>
<p>Basic command to wish you a good day!
I decided that a command doesn’t extends any class because in my opinion is a good way to impart readability and simplicity.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">App\Command</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">ZF\Console\Route</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Zend\Console\Adapter\AdapterInterface</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">Hello</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">static</span> <span class="k">function</span> <span class="nf">run</span><span class="p">(</span><span class="nx">Route</span> <span class="nv">$route</span><span class="p">,</span> <span class="nx">AdapterInterface</span> <span class="nv">$console</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$name</span> <span class="o">=</span> <span class="nv">$route</span><span class="o">-&gt;</span><span class="na">getMatchedParam</span><span class="p">(</span><span class="s2">"name"</span><span class="p">,</span> <span class="s2">"@gianarb"</span><span class="p">);</span>
<span class="nv">$console</span><span class="o">-&gt;</span><span class="na">writeLine</span><span class="p">(</span><span class="s2">"Hi </span><span class="si">{</span><span class="nv">$name</span><span class="si">}</span><span class="s2">, you have call me. Now this is an awesome day!"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<h2 id="troubleshooting-and-tricks">Troubleshooting and tricks</h2>
<ul>
<li>OSx return an error because zf-console use a function blocked into the mac os php installation. Have a look at PR<a href="https://github.com/zfcampus/zf-console/pull/22">#22</a></li>
<li>See <a href="http://www.sitepoint.com/packaging-your-apps-with-phar/">this</a> article to package your application in a phar archive..</li>
</ul>
<p><br />
<br />
<br /></p>
<div class="well"><a target="_blank" href="https://twitter.com/__debo">@__debo</a> thanks for trying to fix my bad English</div>
Test your Symfony Controller and your service with PhpUnit2015-05-21T23:08:27+00:00http://gianarb.it/blog/symfony-unit-test-controller-with-phpunit<blockquote align="center" class="twitter-tweet" lang="en"><p lang="en" dir="ltr">Unit <a href="https://twitter.com/hashtag/test?src=hash">#test</a> for your <a href="https://twitter.com/hashtag/Controller?src=hash">#Controller</a> with <a href="https://twitter.com/hashtag/PhpUnit?src=hash">#PhpUnit</a> and <a href="https://twitter.com/hashtag/Symfony?src=hash">#Symfony</a>.. With a little use case of <a href="https://twitter.com/hashtag/DepedenceInjaction?src=hash">#DepedenceInjaction</a> test <a href="http://t.co/JNb39EyRly">http://t.co/JNb39EyRly</a> <a href="https://twitter.com/hashtag/php?src=hash">#php</a></p>&mdash; Gianluca Arbezzano (@GianArb) <a href="https://twitter.com/GianArb/status/601526550438215680">May 21, 2015</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
<p>In this article I would like share with you a little experience with:</p>
<ul>
<li>Symfony MVC</li>
<li>PhpUnit</li>
<li>Symfony Dependence Injaction</li>
</ul>
<p>This is an example of very easy controller.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">AppBundle\Controller</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Sensio\Bundle\FrameworkExtraBundle\Configuration\Route</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Symfony\Bundle\FrameworkBundle\Controller\Controller</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Symfony\Component\HttpFoundation\Request</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">SomeStuffController</span> <span class="k">extends</span> <span class="nx">FOSRestController</span>
<span class="p">{</span>
<span class="sd">/**
* @Rest\Post("/go")
* @return array
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">goAction</span><span class="p">(</span><span class="nx">Request</span> <span class="nv">$request</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">container</span><span class="o">-&gt;</span><span class="na">getParameter</span><span class="p">(</span><span class="s2">"do_stuff"</span><span class="p">))</span> <span class="p">{</span>
<span class="nv">$body</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">container</span><span class="o">-&gt;</span><span class="na">get</span><span class="p">(</span><span class="s2">"stuff.service"</span><span class="p">)</span><span class="o">-&gt;</span><span class="na">splash</span><span class="p">(</span><span class="nv">$request</span><span class="o">-&gt;</span><span class="na">getContent</span><span class="p">());</span>
<span class="p">}</span>
<span class="k">return</span> <span class="p">[];</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p><code class="highlighter-rouge">$this-&gt;container-&gt;getParameter("do_stuff")</code> is a boolean parameter that enable or disable a feature, How can I test this snippet?
I can try to write a functional test but in my opinion is easier write a series of unit tests with PhpUnit to validate my expectations.</p>
<div class="post row">
<div class="col-md-12">
<a href="http://scaledocker.com?from=gianarb" target="_blank"><img src="http://scaledocker.com/adv/leaderboard.gif" /></a>
</div>
</div>
<h2 id="expectations">Expectations</h2>
<ul>
<li>If <code class="highlighter-rouge">do_stuff</code> parameter is false function get by my container will be call zero times</li>
<li>If <code class="highlighter-rouge">do_stuff</code> parameter is true function get by my container will be call one times</li>
</ul>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">AppBundle\Tests\Controller</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Liip\FunctionalTestBundle\Test\WebTestCase</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">AppBundle\Controller\SomeStuffController</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">SomeStuffControllerTest</span> <span class="k">extends</span> <span class="nx">WebTestCase</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">testDoStuffIsTrue</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$request</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">(</span><span class="s2">"Symfony\Component\HttpFoundation\Request"</span><span class="p">);</span>
<span class="nv">$container</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">(</span><span class="s2">"Symfony\Component\DependencyInjection\ContainerInterface"</span><span class="p">);</span>
<span class="nv">$service</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMockBuilder</span><span class="p">(</span><span class="s2">"Some\Stuff"</span><span class="p">)</span><span class="o">-&gt;</span><span class="na">disableOriginalConstructor</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">();</span>
<span class="nv">$container</span><span class="o">-&gt;</span><span class="na">expects</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">once</span><span class="p">())</span>
<span class="o">-&gt;</span><span class="na">method</span><span class="p">(</span><span class="s2">"getParameter"</span><span class="p">)</span>
<span class="o">-&gt;</span><span class="na">with</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">equalTo</span><span class="p">(</span><span class="s1">'do_stuff'</span><span class="p">))</span>
<span class="o">-&gt;</span><span class="na">will</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">returnValue</span><span class="p">(</span><span class="kc">true</span><span class="p">));</span>
<span class="nv">$container</span><span class="o">-&gt;</span><span class="na">expects</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">once</span><span class="p">())</span>
<span class="o">-&gt;</span><span class="na">method</span><span class="p">(</span><span class="s2">"get"</span><span class="p">)</span>
<span class="o">-&gt;</span><span class="na">with</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">equalTo</span><span class="p">(</span><span class="s1">'stuff.service'</span><span class="p">))</span>
<span class="o">-&gt;</span><span class="na">will</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">returnValue</span><span class="p">(</span><span class="nv">$service</span><span class="p">));</span>
<span class="nv">$controller</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SameStuffController</span><span class="p">();</span>
<span class="nv">$controller</span><span class="o">-&gt;</span><span class="na">setContainer</span><span class="p">(</span><span class="nv">$container</span><span class="p">);</span>
<span class="nv">$controller</span><span class="o">-&gt;</span><span class="na">goAction</span><span class="p">(</span><span class="nv">$request</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>This is my first expetection “If <code class="highlighter-rouge">do_stuff</code> param is true I call <code class="highlighter-rouge">stuff.service</code>”.
In this controller I use a few objects, Http\Request, Container and <code class="highlighter-rouge">stuff.service</code> in this example is a <code class="highlighter-rouge">Some\Stuff</code> class.
In the first step I have created one mock for each object.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="nv">$request</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">(</span><span class="s2">"Symfony\Component\HttpFoundation\Request"</span><span class="p">);</span>
<span class="nv">$container</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">(</span><span class="s2">"Symfony\Component\DependencyInjection\ContainerInterface"</span><span class="p">);</span>
<span class="nv">$service</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMockBuilder</span><span class="p">(</span><span class="s2">"Some\Stuff"</span><span class="p">)</span><span class="o">-&gt;</span><span class="na">disableOriginalConstructor</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">();</span></code></pre></figure>
<p>In the second step I have written my first expetctation, “Call only one time function <code class="highlighter-rouge">getParameter</code> from <code class="highlighter-rouge">$container</code> with argument do_stuff and it returns true”.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="nv">$container</span><span class="o">-&gt;</span><span class="na">expects</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">once</span><span class="p">())</span>
<span class="o">-&gt;</span><span class="na">method</span><span class="p">(</span><span class="s2">"getParameter"</span><span class="p">)</span>
<span class="o">-&gt;</span><span class="na">with</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">equalTo</span><span class="p">(</span><span class="s1">'do_stuff'</span><span class="p">))</span>
<span class="o">-&gt;</span><span class="na">will</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">returnValue</span><span class="p">(</span><span class="kc">true</span><span class="p">));</span></code></pre></figure>
<p>Thanks at this definitions I know that there will be another effect, my action will call only one time <code class="highlighter-rouge">$container-&gt;get("stuff.service")</code> and it will be return an Some\Stuff object.</p>
<p>The second test that we can write is “if <code class="highlighter-rouge">do_stuff</code> is false <code class="highlighter-rouge">$contaner-&gt;get("stuff.service")</code> it will not be called.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">testDoStuffIsFalse</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$request</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">(</span><span class="s2">"Symfony\Component\HttpFoundation\Request"</span><span class="p">);</span>
<span class="nv">$container</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">(</span><span class="s2">"Symfony\Component\DependencyInjection\ContainerInterface"</span><span class="p">);</span>
<span class="nv">$service</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getMockBuilder</span><span class="p">(</span><span class="s2">"Some\Stuff"</span><span class="p">)</span><span class="o">-&gt;</span><span class="na">disableOriginalConstructor</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">getMock</span><span class="p">();</span>
<span class="nv">$container</span><span class="o">-&gt;</span><span class="na">expects</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">once</span><span class="p">())</span>
<span class="o">-&gt;</span><span class="na">method</span><span class="p">(</span><span class="s2">"getParameter"</span><span class="p">)</span>
<span class="o">-&gt;</span><span class="na">with</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">equalTo</span><span class="p">(</span><span class="s1">'do_stuff'</span><span class="p">))</span>
<span class="o">-&gt;</span><span class="na">will</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">returnValue</span><span class="p">(</span><span class="kc">false</span><span class="p">));</span>
<span class="nv">$container</span><span class="o">-&gt;</span><span class="na">expects</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">never</span><span class="p">())</span>
<span class="o">-&gt;</span><span class="na">method</span><span class="p">(</span><span class="s2">"get"</span><span class="p">)</span>
<span class="o">-&gt;</span><span class="na">with</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">equalTo</span><span class="p">(</span><span class="s1">'stuff.service'</span><span class="p">))</span>
<span class="o">-&gt;</span><span class="na">will</span><span class="p">(</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">returnValue</span><span class="p">(</span><span class="nv">$service</span><span class="p">));</span>
<span class="nv">$controller</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">SameStuffController</span><span class="p">();</span>
<span class="nv">$controller</span><span class="o">-&gt;</span><span class="na">setContainer</span><span class="p">(</span><span class="nv">$container</span><span class="p">);</span>
<span class="nv">$controller</span><span class="o">-&gt;</span><span class="na">goAction</span><span class="p">(</span><span class="nv">$request</span><span class="p">);</span>
<span class="p">}</span></code></pre></figure>
The price of modularity2015-02-21T23:08:27+00:00http://gianarb.it/blog/the-price-of-modularity<p>Today all frameworks are <strong>modulable</strong> but it isn’t just a beautiful word, behind it there are a lot of concepts and ideas:</p>
<ul>
<li>The modularity helps you to reuse parts of code in different projects</li>
<li>Every component is indipendent so you work on single part of code</li>
<li>Every <strong>component</strong> solves a specific problem… it’s a beautiful concept that helps you with maintainance!</li>
<li>other stuffs..</li>
</ul>
<p>As you can imagine there is a drawback, all this requires a big effort.
Ideally every component requires personal circle of release, repository, commits, pull requests, travis conf, documentation etc. etc.</p>
<p>Anyway several shorcuts are available. For instance, <em>git subtree</em> could help you in this war but the key is this: you need an agreement to win.</p>
<p>Zend Framwork Community choose another street, <code class="highlighter-rouge">Zend\Mvc</code> in this moment required:</p>
<figure class="highlight"><pre><code class="language-json" data-lang="json"><span class="p">{</span><span class="w">
</span><span class="s2">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"zendframework/zend-mvc"</span><span class="p">,</span><span class="w">
</span><span class="s2">"..."</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="p">,</span><span class="w">
</span><span class="s2">"target-dir"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Zend/Mvc"</span><span class="p">,</span><span class="w">
</span><span class="s2">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"php"</span><span class="p">:</span><span class="w"> </span><span class="s2">"&gt;=5.3.23"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-eventmanager"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-servicemanager"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-form"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-stdlib"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"require-dev"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"zendframework/zend-authentication"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-console"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-di"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-filter"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-http"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-i18n"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-inputfilter"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-json"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-log"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-modulemanager"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-session"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-serializer"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-text"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-uri"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-validator"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-view"</span><span class="p">:</span><span class="w"> </span><span class="s2">"self.version"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"suggest"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"zendframework/zend-authentication"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Zend</span><span class="se">\\</span><span class="s2">Authentication component for Identity plugin"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-config"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Zend</span><span class="se">\\</span><span class="s2">Config component"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-console"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Zend</span><span class="se">\\</span><span class="s2">Console component"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-di"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Zend</span><span class="se">\\</span><span class="s2">Di component"</span><span class="p">,</span><span class="w">
</span><span class="s2">"zendframework/zend-filter"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Zend</span><span class="se">\\</span><span class="s2">Filter component"</span><span class="p">,</span><span class="w">
</span><span class="s2">"..."</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="s2">"..."</span><span class="p">:</span><span class="w"> </span><span class="s2">"..."</span><span class="w">
</span><span class="p">}</span></code></pre></figure>
<p>A few <code class="highlighter-rouge">require-dev</code> dependencies are used into the component to run some features, why? This force me to think <em>“Dependencies of this feature are included or not?”</em>!!
Composer was born to solve it! In my opinion the cost of the question is highest than download a few unused classes.
There are a lot of unused classes? Maybe too much?</p>
<p>Even if the right answer donsn’t exist I think thant some indicators may help you to understand when is the moment to split the component:</p>
<ul>
<li>List of dependencies</li>
<li>Complexity of component</li>
<li>Features</li>
<li>..</li>
</ul>
<p>No shortcuts.</p>
Zend Framework release 2.3.42015-01-14T00:00:00+00:00http://gianarb.it/blog/zf2-release-234<p>Zend Framework 2.3.4 is ready! After 4 mouths the new path version of ZF2 is
published.</p>
<p>How all path release there aren’t new important feature but the list of <a href="https://github.com/zendframework/zf2/pulls?q=is%3Aclosed+is%3Apr+milestone%3A2.3.4+">pull
requests</a>
is very long.</p>
<ul>
<li><a href="https://github.com/zendframework/zf2/pull/7112">#7112</a> You can find into /resources directory a ZF official logo</li>
<li><a href="https://github.com/zendframework/zf2/pull/7087">#7087</a> Happy new year by ZF!</li>
<li><a href="https://github.com/zendframework/zf2/issues/6673">#6673</a> <code class="highlighter-rouge">Zend\Http\Header</code> now support DateTime format for expire Cookie</li>
</ul>
<p>Zend Framework follow <a href="http://semver.org/">semver</a> directives, <code class="highlighter-rouge">2.3.4</code> is a path
release in this version there are a log list of <a href="https://github.com/zendframework/zf2/pulls?q=is%3Aclosed+is%3Apr+milestone%3A2.3.4+label%3Abug">bug
fixes</a></p>
<p>Good download <a href="https://github.com/zendframework/zf2/releases/tag/release-2.3.4">Zend Femework
2.3.4</a>, this is
the
<a href="https://github.com/zendframework/zf2/blob/18534b6f2c14f52898bb208932fedacd5324be63/CHANGELOG.md">changelog</a></p>
Influx DB and PHP implementation2015-01-06T00:00:00+00:00http://gianarb.it/blog/InfluxDB-and-PHP<p>Influx DB is <a href="http://en.wikipedia.org/wiki/Time_series_database">time series
database</a> written in Go.</p>
<p>It supports SQL like queries and it has different entry points, REST API (tcp
protocol) and UDP.</p>
<div class="row">
<div class="col-md-4 col-md-offset-3"><img class="img-fluid" src="/img/influxdb.png" /></div>
</div>
<p>We wrote a <a href="https://github.com/corley/influxdb-php-sdk">sdk</a> to manage
integration between Influx and PHP.</p>
<p>It supports Guzzle Adapter but if you use Zend\Client you can write your
implementation.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="nv">$guzzle</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">\GuzzleHttp\Client</span><span class="p">();</span>
<span class="nv">$options</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Options</span><span class="p">();</span>
<span class="nv">$adapter</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">GuzzleAdapter</span><span class="p">(</span><span class="nv">$guzzle</span><span class="p">,</span> <span class="nv">$options</span><span class="p">);</span>
<span class="nv">$client</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Client</span><span class="p">();</span>
<span class="nv">$client</span><span class="o">-&gt;</span><span class="na">setAdapter</span><span class="p">(</span><span class="nv">$adapter</span><span class="p">);</span></code></pre></figure>
<p>In this case we are using a Guzzle Client, we communicate with Influx in TPC, but we can speak with it in UDP</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="nv">$options</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Options</span><span class="p">();</span>
<span class="nv">$adapter</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">UdpAdapter</span><span class="p">(</span><span class="nv">$options</span><span class="p">);</span>
<span class="nv">$client</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Client</span><span class="p">();</span>
<span class="nv">$client</span><span class="o">-&gt;</span><span class="na">setAdapter</span><span class="p">(</span><span class="nv">$adapter</span><span class="p">);</span></code></pre></figure>
<p>Both of them have the same usage</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="nv">$client</span><span class="o">-&gt;</span><span class="na">mark</span><span class="p">(</span><span class="s2">"app.search"</span><span class="p">,</span> <span class="nv">$points</span><span class="p">,</span> <span class="s2">"s"</span><span class="p">);</span></code></pre></figure>
<p>The first different between udp and tcp is known, TPC after request expects a
response, UDP does not expect anything and in this case does not exist any
delivery guarantee. If you can accept this stuff this is the benchmark:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">Corley<span class="se">\B</span>enchmarks<span class="se">\I</span>nflux DB<span class="se">\A</span>dapterEvent
Method Name Iterations Average Time Ops/second
<span class="nt">------------------------</span> <span class="nt">------------</span> <span class="nt">--------------</span> <span class="nt">-------------</span>
sendDataUsingHttpAdapter: <span class="o">[</span>1,000 <span class="o">]</span> <span class="o">[</span>0.0026700308323] <span class="o">[</span>374.52751]
sendDataUsingUdpAdapter : <span class="o">[</span>1,000 <span class="o">]</span> <span class="o">[</span>0.0000436344147] <span class="o">[</span>22,917.69026]</code></pre></figure>
Zf2 Event, base use2013-11-21T12:38:27+00:00http://gianarb.it/blog/Zf2-Event-base-use<p>Hi! Some mouths ago I have writted a gist for help me to remember a base use of
Events and Event Manager into Zend Fremework, in this article I report this
small tutorial.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">require_once</span> <span class="nx">__DIR__</span><span class="o">.</span><span class="s2">"/vendor/autoload.php"</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">Foo</span>
<span class="p">{</span>
<span class="cm">/* @var \Zend\EventManager\EventManagerInterface */</span>
<span class="k">protected</span> <span class="nv">$eventManager</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getEventManager</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="nv">$this</span><span class="o">-&gt;</span><span class="na">eventManager</span> <span class="nx">instanceof</span> <span class="nx">\Zend\EventManager\EventManagerInterface</span><span class="p">){</span>
<span class="nv">$this</span><span class="o">-&gt;</span><span class="na">eventManager</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">\Zend\EventManager\EventManager</span><span class="p">();</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">eventManager</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">echoHello</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getEventManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">trigger</span><span class="p">(</span><span class="nx">__FUNCTION__</span><span class="o">.</span><span class="s2">"_pre"</span><span class="p">,</span> <span class="nv">$this</span><span class="p">);</span>
<span class="k">echo</span> <span class="s2">"Hello"</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getEventManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">trigger</span><span class="p">(</span><span class="nx">__FUNCTION__</span><span class="o">.</span><span class="s2">"_post"</span><span class="p">,</span> <span class="nv">$this</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$foo</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Foo</span><span class="p">();</span>
<span class="nv">$foo</span><span class="o">-&gt;</span><span class="na">getEventManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">attach</span><span class="p">(</span><span class="s1">'echoHello_pre'</span><span class="p">,</span> <span class="k">function</span><span class="p">(</span><span class="nv">$e</span><span class="p">){</span>
<span class="k">echo</span> <span class="s2">"Wow! "</span><span class="p">;</span>
<span class="p">});</span>
<span class="nv">$foo</span><span class="o">-&gt;</span><span class="na">getEventManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">attach</span><span class="p">(</span><span class="s1">'echoHello_post'</span><span class="p">,</span> <span class="k">function</span><span class="p">(</span><span class="nv">$e</span><span class="p">){</span>
<span class="k">echo</span> <span class="s2">". This example is very good! </span><span class="se">\n</span><span class="s2">"</span><span class="p">;</span>
<span class="p">});</span>
<span class="nv">$foo</span><span class="o">-&gt;</span><span class="na">getEventManager</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">attach</span><span class="p">(</span><span class="s1">'echoHello_post'</span><span class="p">,</span> <span class="k">function</span><span class="p">(</span><span class="nv">$e</span><span class="p">){</span>
<span class="k">echo</span> <span class="s2">"</span><span class="se">\n</span><span class="s2">by gianarb92@gmail.com </span><span class="se">\n</span><span class="s2">"</span><span class="p">;</span>
<span class="p">},</span> <span class="o">-</span><span class="mi">10</span><span class="p">);</span>
<span class="nv">$foo</span><span class="o">-&gt;</span><span class="na">echoHello</span><span class="p">();</span></code></pre></figure>
<p>The result:</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash">gianarb@GianArb-2 eventTest :<span class="o">)</span> <span class="nv">$ </span>php try.php
Wow! Hello. This example is very good!
by gianarb92@gmail.com</code></pre></figure>
<p><a href="http://framework.zend.com/manual/2.0/en/modules/zend.event-manager.event-manager.html">@see Zend Event Manager Ref</a></p>
Git global gitignore2013-11-21T12:38:27+00:00http://gianarb.it/blog/Git-globa-gitignore<p><code class="highlighter-rouge">.gitignore </code> helps me manage my commits by setting which files or
directory don’t end in my repository. I know two good practices if you work for
example on an open source project:</p>
<ul>
<li>You don’t commit your IDE configurations</li>
<li>Not use .gitignore file for exclude IDE configuration, because this is
personal problem. There are differents IDE, if all devs exclude this files on
a repository level the lists is very long.</li>
</ul>
<p>I follow this practices for all my projects, if you are Mac user you have a
DS_STORE files, there is a method for exclude this file of default.</p>
<p><code class="highlighter-rouge">~./.gitconfig</code> is your configuration file, every user has it. If you execute this command</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nv">$.</span> git config <span class="nt">--global</span> core.excludesfile ~/.gitignore_global</code></pre></figure>
<p>into this file it write thiese lines</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[</span>core]
excludesfile <span class="o">=</span> /Users/gianarb/.gitignore_global</code></pre></figure>
<p><code class="highlighter-rouge">/Users/gianarb/.gitignore_global</code> is my global gitignore file!</p>
<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># IDE #</span>
<span class="c">#######</span>
.idea
<span class="c"># COMPOSER #</span>
<span class="c">############</span>
composer.phar
<span class="c"># OS generated files #</span>
<span class="c">######################</span>
.DS_Store
.DS_Store?
._<span class="k">*</span>
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db</code></pre></figure>
Vagrant Up, slide and first talk2013-09-14T12:08:27+00:00http://gianarb.it/blog/vagrant-up-talk-milano<h3>Vagrant Up</h3>
<p>Friday 12 Sept 2013 I talked at Vagrant, tool for manage VM, these is my slide.
I thanks <a href="http://milano.grusp.org/" target="_blank">PugMi</a> for this opportunity if you have questions I'm here! :grin: </p>
<iframe style="display:block; margin: 0 auto;" src="http://www.slideshare.net/slideshow/embed_code/26159972" width="597" height="486" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC;border-width:1px 1px 0;margin-bottom:5px" allowfullscreen webkitallowfullscreen mozallowfullscreen> </iframe> <div style="margin-bottom:5px;text-align:center;"> <strong> <a href="https://www.slideshare.net/GianlucaArbezzano/presentazione-def-26159972" title="Vagrant - PugMI" target="_blank">Vagrant - PugMI</a> </strong> from <strong><a href="http://www.slideshare.net/GianlucaArbezzano" target="_blank">Gianluca Arbezzano</a></strong> </div>
</br></br>
<blockquote class="twitter-tweet" align="center"><p>quasi 30 persone a sentire <a href="https://twitter.com/GianArb">
@GianArb</a> parlare di <a href="https://twitter.com/search?q=%23vagrant&amp;src=hash">#vagrant</a> <a href="https://twitter.com/search?q=%23php&amp;src=hash">#php</a> <a href="https://twitter.com/search?q=%23pugMi&amp;src=hash">#pugMi</a> <a href="https://twitter.com/search?q=%23milano&amp;src=hash">#milano</a> <a href="http://t.co/75MOJiZmDZ">pic.twitter.com/75MOJiZmDZ</a></p>&mdash; Milano PHP (@MilanoPHP) <a href="https://twitter.com/MilanoPHP/statuses/378213865072656385">September 12, 2013</a></blockquote>
<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
Zend Framework 2 - Console usage a speed help2013-08-22T08:08:27+00:00http://gianarb.it/blog/zf2-console-usage-speed-help<p>With Zend Framework is very easy to write a command line tool to manage
different things. But what if there are more commands? How do you remeber them
all?</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">ModuleTest</span><span class="p">;</span>
<span class="k">use</span> <span class="nx">Zend\Console\Adapter\AdapterInterface</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">Module</span> <span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">getConsoleUsage</span><span class="p">(</span><span class="nx">AdapterInterface</span> <span class="nv">$console</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">array</span><span class="p">(</span>
<span class="k">array</span><span class="p">(</span><span class="s1">'test &lt;params1&gt; &lt;params2&gt; [--params=]'</span><span class="p">,</span> <span class="s1">'Description of test command'</span><span class="p">),</span>
<span class="k">array</span><span class="p">(</span><span class="s1">'run &lt;action&gt;'</span><span class="p">,</span> <span class="s1">'Start anction'</span><span class="p">)</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>You can write this function in a Module.php file, and create a basic helper to
help you see when you write a bad command.</p>
<p>English by Rali :smile: Thanks!!!! :smile:</p>
Generale Jekyll sitemap without plugin2013-08-09T09:38:27+00:00http://gianarb.it/blog/Generate-jekyll-sitemap-without-plugin<p>This blog is a static blog and uses GitHub pages, GitHub pages are generally
deployed using Jekyll.</p>
<h3 id="how-can-you-generate-a-sitemap-without-jekyll-plugin">How can you generate a sitemap without Jekyll plugin?</h3>
<p>This <a href="https://gist.github.com/GianArb/6172377">gist</a> answers your question.</p>
<p>I use some post values: changefreq, date and priority, if you don’t set any
specific values for them default values are used that are, 0.8 for priority and
month for frequency.</p>
<p>In a single post you add this params for use correct params!</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="o">---</span>
<span class="nx">layout</span><span class="o">:</span> <span class="nx">post</span>
<span class="nx">title</span><span class="o">:</span> <span class="s2">"Why this blog?"</span>
<span class="nb">date</span><span class="o">:</span> <span class="mi">2013</span><span class="o">-</span><span class="mo">07</span><span class="o">-</span><span class="mi">22</span> <span class="mi">23</span><span class="o">:</span><span class="mi">08</span><span class="o">:</span><span class="mi">27</span>
<span class="nx">categories</span><span class="o">:</span> <span class="nx">me</span>
<span class="nx">tags</span><span class="o">:</span> <span class="nx">me</span><span class="p">,</span> <span class="nx">developer</span><span class="p">,</span> <span class="nx">presentation</span><span class="p">,</span> <span class="nx">gianarb</span>
<span class="nx">summary</span><span class="o">:</span> <span class="nx">Gianluca</span> <span class="nx">Arbezzano</span><span class="p">,</span> <span class="nx">developer</span><span class="p">,</span> <span class="nx">Italian</span><span class="p">,</span> <span class="nx">why</span> <span class="nx">open</span> <span class="k">this</span> <span class="nx">blog</span><span class="o">?</span>
<span class="nx">changefreq</span><span class="o">:</span> <span class="nx">monthly</span>
<span class="o">---</span></code></pre></figure>
<p>If you want to know more about the Sitemap Protocol read
<a href="http://www.sitemaps.org/protocol.html">this</a>.</p>
<p><a href="https://github.com/MarcoDeBortoli">Marco</a> thanks for English! :)</p>
Zend Framework 2 - How do you implement log service?2013-07-26T23:08:27+00:00http://gianarb.it/blog/how-do-you-implement-log-service<p>A log system is an essential element for any application. It is a way to check
the status and use of the application. For a basic implementation you can refer
to the fig-stanrdars organization
<a href="https://github.com/php-fig/fig-standards/blob/master/accepted/PSR-3-logger-interface.md">PSR-3</a>
article, that describes th elogger interface.</p>
<p>Zend Framework 2 implement a <a href="https://github.com/zendframework/zf2/tree/master/library/Zend/Log">Logger
Component</a>,
the following is an example of how to use it with service manager.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">return</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'service_manager'</span> <span class="o">=&gt;</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'abstract_factories'</span> <span class="o">=&gt;</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'Zend\Log\LoggerAbstractServiceFactory'</span><span class="p">,</span>
<span class="p">),</span>
<span class="p">),</span>
<span class="s1">'log'</span> <span class="o">=&gt;</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'Log\App'</span> <span class="o">=&gt;</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'writers'</span> <span class="o">=&gt;</span> <span class="k">array</span><span class="p">(</span>
<span class="k">array</span><span class="p">(</span>
<span class="s1">'name'</span> <span class="o">=&gt;</span> <span class="s1">'stream'</span><span class="p">,</span>
<span class="s1">'priority'</span> <span class="o">=&gt;</span> <span class="mi">1000</span><span class="p">,</span>
<span class="s1">'options'</span> <span class="o">=&gt;</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'stream'</span> <span class="o">=&gt;</span> <span class="s1">'data/app.log'</span><span class="p">,</span>
<span class="p">),</span>
<span class="p">),</span>
<span class="p">),</span>
<span class="p">),</span>
<span class="p">),</span>
<span class="p">);</span></code></pre></figure>
<p><a href="https://github.com/zendframework/zf2/blob/master/library/Zend/Log/LoggerServiceFactory.php">LoggerAbstractServiceFactory</a>
is a Service Factory, as an example, into service Manager class Logger and will
be used in the whole application. Log/App is the name of a single logger, and
writer is an adapter that is used to choose the method of writing, in this case
everything is written to file, but you can use a DB adapter and write your log
into database.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">namespace</span> <span class="nx">GianArb\Controller</span><span class="p">;</span>
<span class="k">class</span> <span class="nc">GeneralController</span>
<span class="k">extends</span> <span class="nx">AbastractActionController</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="nf">testAction</span><span class="p">(){</span>
<span class="nv">$logger</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-&gt;</span><span class="na">getServiceLocator</span><span class="p">()</span><span class="o">-&gt;</span><span class="na">get</span><span class="p">(</span><span class="s1">'Log\App'</span><span class="p">);</span>
<span class="nv">$logger</span><span class="o">-&gt;</span><span class="na">log</span><span class="p">(</span><span class="nx">\Zend\Log\Logger</span><span class="o">::</span><span class="na">INFO</span><span class="p">,</span> <span class="s2">"This is a little log!"</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>With this configuration Log\App writes a string into data/app.log file, with
INFO property. By default you can use an array of properties.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">protected</span> <span class="nv">$priorities</span> <span class="o">=</span> <span class="k">array</span><span class="p">(</span>
<span class="nx">self</span><span class="o">::</span><span class="na">EMERG</span> <span class="o">=&gt;</span> <span class="s1">'EMERG'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">ALERT</span> <span class="o">=&gt;</span> <span class="s1">'ALERT'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">CRIT</span> <span class="o">=&gt;</span> <span class="s1">'CRIT'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">ERR</span> <span class="o">=&gt;</span> <span class="s1">'ERR'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">WARN</span> <span class="o">=&gt;</span> <span class="s1">'WARN'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">NOTICE</span> <span class="o">=&gt;</span> <span class="s1">'NOTICE'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">INFO</span> <span class="o">=&gt;</span> <span class="s1">'INFO'</span><span class="p">,</span>
<span class="nx">self</span><span class="o">::</span><span class="na">DEBUG</span> <span class="o">=&gt;</span> <span class="s1">'DEBUG'</span><span class="p">,</span>
<span class="p">);</span></code></pre></figure>
<p>Usage of different keys is a good practice because it is very easy to write
filter or log categories.</p>
<p>Another good practice, valid for all services in general, is to create your
class extending single service.</p>
<figure class="highlight"><pre><code class="language-php" data-lang="php"><span class="cp">&lt;?php</span>
<span class="k">use</span> <span class="nx">Zend\Log\Logger</span>
<span class="k">class</span> <span class="nc">MyLogger</span> <span class="k">extends</span> <span class="nx">Logger</span></code></pre></figure>
<p>This choice helps managing future customizations of services and is another
important layer for managing unexpected updates.</p>
<p>Rali, thanks for your help with my robotic english! :P</p>
Why this blog?2013-07-22T23:08:27+00:00http://gianarb.it/blog/why-this-blog<p>Hi! I’m Gianluca aka GianArb, I’m web developer, works with Php, SQL and noSql
databases and in this time I’m crazy for DevOps, Vagrant and Chef, than for
manage this tool I’m learning Ruby.</p>
<h3 id="why-this-blog">Why this blog?</h3>
<p>I’m opening this blog because my English is very terrible! I have an italian
<a href="http://gianarb.it">Blog</a> in wordpress but I’d like to use Jekyll and this is a
good opportunity. Share my experience and my Job, to grow and improve my
skills! Can you help me with my English? :P</p>
<h3 id="skills-and-interests">Skills and interests</h3>
<p>This is a list of my skills and interest and are probabily topics for my posts,
I hope you like it! Php, tech, html, css, js, Open Source, Zf2, Doctrine,
Symfony, noSql (mongo, couch..), sql databases, Redis, DevOps, Chef, Vagrant,
Composer, TDD….</p>
<h3 id="open-source-world">Open Source world!</h3>
<p>Community and Open Source are my pessions! Free code is a good method for to put
oneself to the test, my street is very long but this is my <a href="http://github.com/GianArb">Github
Account</a>!</p>
<h3 id="this-is-my-face">This is my face!</h3>
<div style="text-align:center;">
<img src="/img/posts/2013-07-19-why-this-blog.png" width="90%" />
</div>