Elixirator Bloghttp://elixirator.com/blog2017-08-15T00:00:00+03:00ElixiratorHow to create Slack bot in Elixirhttp://elixirator.com/blog/2017/elixir-slack-bot.html2017-08-15T00:00:00+03:002017-10-31T17:52:25+02:00Alex Beznos<p>In our company Elixirator, we have an investment time every Friday. Four hours of free time when anyone can build tools that can improve or facilitate our everyday workflow. As my project, I decided to build the Slack bot, that will help to store every day notes and remind about them before the weekly meeting. In this article I will show how to build a small slack bot via Elixir lang and I will point your attention to the little things that were not described in the official documentation.</p>
<p></p><p>In our company Elixirator, we have an investment time every Friday. Four hours of free time when anyone can build tools that can improve or facilitate our everyday workflow. As my project, I decided to build the Slack bot, that will help to store every day notes and remind about them before the weekly meeting. In this article I will show how to build a small slack bot via Elixir lang and I will point your attention to the little things that were not described in the official documentation.</p>
<p></p>
<h1>Introduction</h1>
<p>First of all, after the decision to create a bot, I had to decide which language to choose. At Elixirator I work as a ruby developer, so I flatly refused to use ruby lang for it. After that, my attention was caught by <a href="https://hubot.github.com/">Hubot</a> and javascript, but to write a code in Coffeescript in 2k17, come on 😂. So I chose Elixir with <a href="https://github.com/hedwig-im/hedwig">Hedwig</a> chat-bot framework.
As my main stack I will use such libs:</p>
<ul>
<li><a href="https://github.com/hedwig-im/hedwig">Hedwig</a> - as a chat bot framework</li>
<li><a href="https://github.com/elixir-ecto/ecto">Ecto</a> - as a database toolkit.</li>
<li><a href="https://github.com/elixir-ecto/postgrex">Postgrex</a> - PostgreSQL adapter for Ecto</li>
<li><a href="https://github.com/jbernardo95/cronex">Cronex</a> - simple scheduler inspired by well-known <a href="https://github.com/javan/whenever">whenever</a> Ruby gem.</li>
</ul>
<h1>Expectation</h1>
<p>Our bot should be able to accept messages from the user during the week, aggregate it and remind about them in 15 minutes till the weekly meeting. Also it should be able to explicitly give back messages added during the week and user should be able to remove any of them. Expected chating should be like that:</p>
<div class="highlight"><pre class="highlight plaintext"><code>user: bot note add more specs to the current task
bot: Get it
user: bot note send info mail to the customer
bot: Get it
user: bot notes list
bot:
1 - add more specs to the current task
2 - send info mail to the customer
Check it out.
user: bot destroy note 2
bot: I threw it away
</code></pre></div>
<h1>Setup</h1>
<p>To create an application in Elixir for Hedwig chat-bot you need to run:</p>
<div class="highlight"><pre class="highlight shell"><code><span class="nv">$ </span>mix new bot_name <span class="nt">--sup</span>
</code></pre></div>
<p>The <code>sup</code> option is given to generate an OTP application with supervision tree. We need an supervisor to deal with our Hedwig bot, Ecto and scheduler processes.
After the creation of the application we need to add required libraries to the dependencies:</p>
<div class="highlight"><pre class="highlight elixir"><code><span class="c1"># mix.exs file</span>
<span class="k">def</span> <span class="n">application</span> <span class="k">do</span>
<span class="p">[</span><span class="ss">extra_applications:</span> <span class="p">[</span><span class="ss">:hedwig</span><span class="p">,</span> <span class="ss">:ecto</span><span class="p">,</span> <span class="ss">:postgrex</span><span class="p">,</span> <span class="ss">:logger</span><span class="p">],</span>
<span class="ss">mod:</span> <span class="p">{</span><span class="no">BotName</span><span class="o">.</span><span class="no">Application</span><span class="p">,</span> <span class="p">[]}]</span>
<span class="k">end</span>
<span class="k">defp</span> <span class="n">deps</span> <span class="k">do</span>
<span class="p">[</span>
<span class="p">{</span><span class="ss">:hedwig</span><span class="p">,</span> <span class="sd">"</span><span class="s2">~&gt; 1.0"</span><span class="p">},</span>
<span class="p">{</span><span class="ss">:postgrex</span><span class="p">,</span> <span class="sd">"</span><span class="s2">&gt;= 0.0.0"</span><span class="p">},</span>
<span class="p">{</span><span class="ss">:ecto</span><span class="p">,</span> <span class="sd">"</span><span class="s2">~&gt; 2.1"</span><span class="p">},</span>
<span class="p">{</span><span class="ss">:cronex</span><span class="p">,</span> <span class="sd">"</span><span class="s2">~&gt; 0.4.0"</span><span class="p">}</span>
<span class="p">]</span>
<span class="k">end</span>
</code></pre></div>
<p>To fetch them run:</p>
<div class="highlight"><pre class="highlight shell"><code><span class="nv">$ </span>mix deps.get
</code></pre></div>
<p>After the fetching, you need to generate the bot module by the command <code>mix hedwig.gen.robot</code> and add it to your supervision tree.</p>
<div class="highlight"><pre class="highlight elixir"><code><span class="c1"># lib/bot_name/application.ex</span>
<span class="k">defmodule</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Application</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Application</span>
<span class="k">def</span> <span class="n">start</span><span class="p">(</span><span class="n">_type</span><span class="p">,</span> <span class="n">_args</span><span class="p">)</span> <span class="k">do</span>
<span class="kn">import</span> <span class="no">Supervisor</span><span class="o">.</span><span class="no">Spec</span><span class="p">,</span> <span class="ss">warn:</span> <span class="no">false</span>
<span class="n">children</span> <span class="o">=</span> <span class="p">[</span>
<span class="n">worker</span><span class="p">(</span><span class="no">BotName</span><span class="o">.</span><span class="no">Robot</span><span class="p">,</span> <span class="p">[])</span>
<span class="p">]</span>
<span class="n">opts</span> <span class="o">=</span> <span class="p">[</span><span class="ss">strategy:</span> <span class="ss">:one_for_one</span><span class="p">,</span> <span class="ss">name:</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Supervisor</span><span class="p">]</span>
<span class="no">Supervisor</span><span class="o">.</span><span class="n">start_link</span><span class="p">(</span><span class="n">children</span><span class="p">,</span> <span class="n">opts</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>All modules which should be supervised will be added to the <code>children</code> list. Further, we will add here Ecto repository and a scheduler. But for now, we are ready to add our first responder.</p>
<h1>Responders</h1>
<p>For the current application I need three responders:</p>
<ul>
<li>Notes creation</li>
<li>Notes destroy</li>
<li>Notes list</li>
</ul>
<p>All of them were decided to leave in one file because the logic of processing will be leaved in services.
Hedwig provides two macroses: <code>hear/2</code> and <code>respond/2</code>. The difference between them is that <code>hear/2</code> matches messages containing regular expression and <code>respond/2</code> matches only when regular expression is prefixed by the bot name or bot name alias. My responder will looks like that:</p>
<div class="highlight"><pre class="highlight elixir"><code><span class="c1"># lib/bot_name/responders/notes.ex</span>
<span class="k">defmodule</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Responders</span><span class="o">.</span><span class="no">Notes</span> <span class="k">do</span>
<span class="kn">use</span> <span class="no">Hedwig</span><span class="o">.</span><span class="no">Responder</span>
<span class="n">alias</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Services</span><span class="o">.</span><span class="no">Notes</span><span class="o">.</span><span class="no">Create</span>
<span class="n">alias</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Services</span><span class="o">.</span><span class="no">Notes</span><span class="o">.</span><span class="no">List</span>
<span class="n">alias</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Services</span><span class="o">.</span><span class="no">Notes</span><span class="o">.</span><span class="no">Destroy</span>
<span class="n">alias</span> <span class="no">Hedwig</span><span class="o">.</span><span class="no">Message</span>
<span class="n">hear</span> <span class="sr">~r/note (.*)/</span><span class="p">,</span> <span class="p">%</span><span class="no">Message</span><span class="p">{</span><span class="ss">matches:</span> <span class="p">%{</span><span class="m">1</span> <span class="o">=&gt;</span> <span class="n">content</span><span class="p">},</span> <span class="ss">user:</span> <span class="n">user</span><span class="p">,</span> <span class="ss">room:</span> <span class="n">room</span><span class="p">}</span> <span class="o">=</span> <span class="n">msg</span> <span class="k">do</span>
<span class="n">emote</span> <span class="n">msg</span><span class="p">,</span> <span class="no">Create</span><span class="o">.</span><span class="n">run</span><span class="p">(%{</span><span class="ss">username:</span> <span class="n">user</span><span class="p">,</span> <span class="ss">room:</span> <span class="n">room</span><span class="p">,</span> <span class="ss">content:</span> <span class="n">content</span><span class="p">})</span>
<span class="k">end</span>
<span class="n">hear</span> <span class="sr">~r/notes list/</span><span class="p">,</span> <span class="p">%</span><span class="no">Message</span><span class="p">{</span><span class="ss">user:</span> <span class="n">user</span><span class="p">}</span> <span class="o">=</span> <span class="n">msg</span> <span class="k">do</span>
<span class="n">emote</span> <span class="n">msg</span><span class="p">,</span> <span class="no">List</span><span class="o">.</span><span class="n">run</span><span class="p">(%{</span><span class="ss">username:</span> <span class="n">user</span><span class="p">})</span>
<span class="k">end</span>
<span class="n">hear</span> <span class="sr">~r/destroy (?&lt;all&gt;all) (?=notes)|destroy notes (?&lt;ids&gt;.*)/</span><span class="p">,</span> <span class="n">msg</span> <span class="k">do</span>
<span class="n">emote</span> <span class="n">msg</span><span class="p">,</span> <span class="no">Destroy</span><span class="o">.</span><span class="n">run</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h1>Sensitive data</h1>
<p>To start working on the processing logic, first of all, we need to setup Ecto with Postgrex. It is nicely documented <a href="https://github.com/elixir-ecto/ecto">here</a> so I will not go into details. But this documentation is not covering the topic: how to deal with sensitive data. Because no one wants to store passwords in the repository on Github or wherever. During the research, I haven&rsquo;t found any tool that solves this problem, so had to make it in such way. My configurations file looks like that:</p>
<div class="highlight"><pre class="highlight elixir"><code><span class="c1"># config/config.exs</span>
<span class="kn">use</span> <span class="no">Mix</span><span class="o">.</span><span class="no">Config</span>
<span class="n">config</span> <span class="ss">:slave</span><span class="p">,</span> <span class="ss">ecto_repos:</span> <span class="p">[</span><span class="no">BotName</span><span class="o">.</span><span class="no">Repo</span><span class="p">]</span>
<span class="n">config</span> <span class="ss">:slave</span><span class="p">,</span> <span class="no">BotName</span><span class="o">.</span><span class="no">Repo</span><span class="p">,</span>
<span class="ss">adapter:</span> <span class="no">Ecto</span><span class="o">.</span><span class="no">Adapters</span><span class="o">.</span><span class="no">Postgres</span><span class="p">,</span>
<span class="ss">database:</span> <span class="no">System</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="sd">"</span><span class="s2">DB_NAME"</span><span class="p">),</span>
<span class="ss">username:</span> <span class="no">System</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="sd">"</span><span class="s2">DB_USERNAME"</span><span class="p">),</span>
<span class="ss">password:</span> <span class="no">System</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="sd">"</span><span class="s2">DB_PASS"</span><span class="p">),</span>
<span class="ss">hostname:</span> <span class="no">System</span><span class="o">.</span><span class="n">get_env</span><span class="p">(</span><span class="sd">"</span><span class="s2">DB_HOSTNAME"</span><span class="p">)</span>
<span class="n">import_config</span> <span class="sd">"</span><span class="si">#{</span><span class="no">Mix</span><span class="o">.</span><span class="n">env</span><span class="si">}</span><span class="s2">.exs"</span>
</code></pre></div>
<p>Depending on the environment, it imports different configs which merged with the current one. Also, all sensitive data are taken from environment variables. To expose them to the development environment, <code>config/.env</code> file was made.</p>
<div class="highlight"><pre class="highlight shell"><code><span class="c"># config/.env</span>
<span class="nb">export </span><span class="nv">DB_NAME</span><span class="o">=</span><span class="s2">"bot_name_dev"</span>
<span class="nb">export </span><span class="nv">DB_USERNAME</span><span class="o">=</span><span class="s2">"bot_name_app"</span>
<span class="nb">export </span><span class="nv">DB_PASS</span><span class="o">=</span><span class="s2">"2eYdW8D4"</span>
<span class="nb">export </span><span class="nv">DB_HOSTNAME</span><span class="o">=</span><span class="s2">"localhost"</span>
</code></pre></div>
<p>Before the application run, we should export it with:</p>
<div class="highlight"><pre class="highlight shell"><code><span class="nv">$ </span><span class="nb">source </span>config/.env
</code></pre></div>
<p>After that you are ready to go.</p>
<h1>Gotchas</h1>
<p>Depending on adapter, message has a bit different structure. In console adapter, <code>user</code> will be a string, but in case of Slack adapter, it gonna be a map with user name and user id.</p>
<div class="highlight"><pre class="highlight elixir"><code><span class="c1"># Messages inspected in create `hear` responder:</span>
<span class="c1"># console adapter</span>
<span class="p">%</span><span class="no">Hedwig</span><span class="o">.</span><span class="no">Message</span><span class="p">{</span>
<span class="ss">matches:</span> <span class="p">%{</span><span class="m">0</span> <span class="o">=&gt;</span> <span class="sd">"</span><span class="s2">note list"</span><span class="p">,</span> <span class="m">1</span> <span class="o">=&gt;</span> <span class="sd">"</span><span class="s2">list"</span><span class="p">},</span>
<span class="ss">private:</span> <span class="p">%{},</span>
<span class="ss">ref:</span> <span class="c1">#Reference&lt;0.0.4.1460&gt;,</span>
<span class="ss">robot:</span> <span class="c1">#PID&lt;0.321.0&gt;,</span>
<span class="ss">room:</span> <span class="no">nil</span><span class="p">,</span>
<span class="ss">text:</span> <span class="sd">"</span><span class="s2">note list"</span><span class="p">,</span>
<span class="ss">type:</span> <span class="sd">"</span><span class="s2">chat"</span><span class="p">,</span>
<span class="ss">user:</span> <span class="sd">"</span><span class="s2">beznosa"</span>
<span class="p">}</span>
<span class="c1"># slack adapter</span>
<span class="p">%</span><span class="no">Hedwig</span><span class="o">.</span><span class="no">Message</span><span class="p">{</span>
<span class="ss">matches:</span> <span class="p">%{</span><span class="m">0</span> <span class="o">=&gt;</span> <span class="sd">"</span><span class="s2">note list"</span><span class="p">,</span> <span class="m">1</span> <span class="o">=&gt;</span> <span class="sd">"</span><span class="s2">list"</span><span class="p">},</span>
<span class="ss">private:</span> <span class="p">%{},</span>
<span class="ss">ref:</span> <span class="c1">#Reference&lt;0.0.1.2804&gt;,</span>
<span class="ss">robot:</span> <span class="c1">#PID&lt;0.214.0&gt;,</span>
<span class="ss">room:</span> <span class="sd">"</span><span class="s2">D5ZNBVD2P"</span><span class="p">,</span>
<span class="ss">text:</span> <span class="sd">"</span><span class="s2">note list"</span><span class="p">,</span>
<span class="ss">type:</span> <span class="sd">"</span><span class="s2">message"</span><span class="p">,</span>
<span class="ss">user:</span> <span class="p">%</span><span class="no">Hedwig</span><span class="o">.</span><span class="no">User</span><span class="p">{</span>
<span class="ss">id:</span> <span class="sd">"</span><span class="s2">U0JBHFDK9"</span><span class="p">,</span>
<span class="ss">name:</span> <span class="sd">"</span><span class="s2">beznosa"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<p>This fact is bothering me a bit, because it&rsquo;s not clear how to work in the development and production environment when message structure is different from adapter to adapter (because when I started, I thought to use console adapter in development and the Slack adapter in the production environment).</p>
<h1>Wrapping up</h1>
<p>During the whole development I felt the lack of documentation, tutorials, guides, but it&rsquo;s probably due to the youth of the Elixir lang and youth of libs. To be honest, the source code was the best documentation for me. I hope, it&rsquo;s gonna be changed soon with our efforts.
Results of my work that is done so far can be checked <a href="https://github.com/elixirator/slave_bot">here</a>. In further articles I will describe how to deploy it.</p>
Our daily toolshttp://elixirator.com/blog/2017/our-daily-tools.html2017-08-01T00:00:00+03:002017-08-01T17:38:29+03:00Artem Melokumov<p>While creating an awesome software it’s important to be efficient and have maximum collaboration. There are lot of tools to make developers life easier nowadays, each with it’s own advantages and disadvantages. Today I’d like to share what apps and services we are using daily in our work.
</p><p>While creating an awesome software it’s important to be efficient and have maximum collaboration. There are lot of tools to make developers life easier nowadays, each with it&rsquo;s own advantages and disadvantages. Today I’d like to share what apps and services we are using daily in our work.
</p>
<h5>Communication: <a href="https://slack.com/">Slack</a></h5>
<ul>
<li>Pros: Cool. Convenient chat-rooms. Integrations. File sharing.</li>
<li>Cons: A heavy app client, that occasionally lags.</li>
<li>A tool that became essential for us from first days of our work. We use it to communicate with clients, to chat with team-members, to gather food orders, to track latest tech-tweets, to share links and files. Ability to edit message during 5 minutes. Besides that in paid version the messaging history is saved becomes really handy when you’re looking for some lost file or detail that was approved by client in the past (◕‿◕)</li>
</ul>
<h5>Task management: <a href="https://trello.com">Trello</a></h5>
<ul>
<li>Pros: Easy. Intuitive. Filters!</li>
<li>Cons: When boards have too many cards - it gets clumsy, performance for moving cards, a search is slowed down significantly.</li>
<li>We have looked through many applications that help to manage tasks, and Trello was chosen for its simplicity - new member can set it up and get where we are in couple minutes. File uploads, integrations, mobile app - great benefits to us. Also, clients also can easy track the progress that is really visual when you look at lists and what cards are on it.</li>
</ul>
<h5>Time tracking: <a href="https://everhour.com">Everhour</a></h5>
<ul>
<li>Pros: Customizable reports. Trello integrations. Browser extension. Responsive support.</li>
<li>Cons: Occasional downtimes. No API.</li>
<li>A well-done product that is rapidly developing. We are tracking time on tasks that we perform so trello integration is really handy and you can use a browser extension (Chrome, Firefox, Safari). Beside tracking on the Trello tasks you can create you can set list of tasks per project/team (like “weekly meeting”, “helping teammates”, “investment time” etc).</li>
</ul>
<h5>Bug tracking: <a href="https://appsignal.com">Appsignal</a></h5>
<ul>
<li>Pros: Customizable bug reports. Slack, Trello integrations.</li>
<li>Cons: Can’t assign User right inside the application (so you are not sure if a problem is solving at the moment by some team-member). Takes time to set it up in a right way you need.</li>
<li>Bug reports are coming to dedicated slack-channel so every team-member can check out what has happened.</li>
</ul>
<h5>Continuous integration: <a href="https://circleci.com">CirceCI</a></h5>
<ul>
<li>Pros: Quick setup. Perfect for test suits. Responsive UI.</li>
<li>Cons: Hard to set complex notifications. Upgrades without warnings.</li>
<li>Whenever a new commit is pushed to GitHub, CircleCI runs the tests that have been already defined and if none of them fails gives a green marker for code review.</li>
</ul>
<h5>Deploy: <a href="https://deploybot.com">Deploybot</a></h5>
<ul>
<li>As we primary using XP practices, fast delivery is essential for us. Since someone has merged approved pull-request to Master it can be immediately deployed to Live with help of Deploybot. Decreases bottleneck and keep access to Live server secured.</li>
</ul>
<h5>Mail catcher: <a href="https://mailtrap.io">Mailtrap</a></h5>
<ul>
<li>Pros: Nice UI. Detailed email info.</li>
<li>Cons: Haven’t noticed any</li>
<li>Great, simple integration that can be installed to your staging server to test email services and deliveries. Basically, catches all emails from selected server and shows all needed email details.</li>
</ul>
<h5>Code repository: <a href="https://github.com">Github</a></h5>
<ul>
<li>Pros: Most commonly used among developers with our stack. Open-source libraries</li>
<li>Cons: You have to pay for every private repo.</li>
<li>We are using it for all our projects. And it was built using Ruby on Rails ;)</li>
</ul>
<h5>Google services: <a href="https://mail.google.com">Mail</a>, <a href="https://calendar.google.com">Calendar</a>, <a href="https://drive.google.com">Drive</a>, <a href="https://docs.google.com">Docs</a>.</h5>
<ul>
<li>Pros: Proven. Reliable</li>
<li>Cons: UI (calendar is sooo boring!)</li>
<li>Google ecosystem allows managing all these services pretty easy. Hard to add something here</li>
</ul>
<h5>Money sharing (ಠ‿ಠ) <a href="https://www.splitwise.com/">Splitwise</a></h5>
<ul>
<li>When it comes to ordering some pizza or sushi for lunch - someone pays for it and makes a record in the app who owes him for what. We found Splitwise as an extremely convenient app as it takes out a headache for money splitting and debt control.</li>
</ul>
<h3>Conclusion</h3>
<p>As you probably noticed we pick our tools by principals: simple, integrative, user-friendly, proven and that has responsible support. Do you have in mind other alternatives or tools that can be helpful? Please, share with us.</p>
<p>Peace to all! (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧</p>
Background file uploads to S3 using Shrinehttp://elixirator.com/blog/2017/background-file-uploads-to-s3-using-shrine.html2017-07-21T00:00:00+03:002017-08-01T17:09:05+03:00Oleksii Korondevych<p>An application of our customer was completely paralyzed on February 28 when AWS S3 service had a major outage because the application’s key features rely on upload of an application-generated file to AWS S3. To improve the customer experience we decided to serve files from cache (DB) while the files are being uploaded to S3 in a background job.
</p><p>An application of our customer was completely paralyzed on February 28 when AWS S3 service had a major outage because the application’s key features rely on upload of an application-generated file to AWS S3. To improve the customer experience we decided to serve files from cache (DB) while the files are being uploaded to S3 in a background job.
</p>
<p>We wrote a module that allows any ActiveRecord model to upload a file to the DB cache first and then it uploaded to S3 in the background. We&rsquo;ve added a service that provides either a relative link to an endpoint that serves a file from the database if it wasn&rsquo;t uploaded or S3 URL.</p>
<p>We’ve decided to replace Paperclip by <a href="http://shrinerb.com">Shrine</a> because it can store files in the database out of the box first of all and was designed to work in the background in mind and also it is more flexible and has rich plugin ecosystem.</p>
<h3>What do we need:</h3>
<ol>
<li>Write migrations: one for ActiveRecord model that will add an <code>&lt;attachment_name&gt;_data</code> and <code>uploaded</code> columns, and another for uploader&rsquo;s cache storage.</li>
<li>Set up Shrine storages and create an uploader class, our attachments will be processed here.</li>
<li>Add Job that will upload a file to S3 and bind current record with <code>binary_file</code> where we store our cache.</li>
<li>Add service class that will decide from which storage fetch needed file, either from S3 if it was uploaded, or from the cache.</li>
<li>Write a class-like module that will combine abilities of the uploader and the file retrieving service. The Module we can include in any model that deal with upload process, by passing a name of attachment attribute as an argument.</li>
</ol>
<h3>Write the migrations</h3>
<p>Add a Migration for a model:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="k">class</span> <span class="nc">AddAttributesToTableName</span> <span class="o">&lt;</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Migration</span>
<span class="k">def</span> <span class="nf">change</span>
<span class="n">change_table</span> <span class="ss">:table_name</span> <span class="k">do</span> <span class="o">|</span><span class="n">t</span><span class="o">|</span>
<span class="n">t</span><span class="p">.</span><span class="nf">text</span> <span class="ss">:&lt;</span><span class="n">attribute_name</span><span class="o">&gt;</span><span class="n">_data</span>
<span class="n">t</span><span class="p">.</span><span class="nf">boolean</span> <span class="ss">:uploaded</span><span class="p">,</span> <span class="ss">default: </span><span class="kp">false</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>Add a migration for the storage:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="k">class</span> <span class="nc">CreateBinaryFiles</span> <span class="o">&lt;</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Migration</span>
<span class="k">def</span> <span class="nf">change</span>
<span class="n">create_table</span> <span class="ss">:binary_files</span> <span class="k">do</span> <span class="o">|</span><span class="n">t</span><span class="o">|</span>
<span class="n">t</span><span class="p">.</span><span class="nf">binary</span> <span class="ss">:content</span>
<span class="n">t</span><span class="p">.</span><span class="nf">json</span> <span class="ss">:metadata</span>
<span class="n">t</span><span class="p">.</span><span class="nf">references</span> <span class="ss">:attachable</span><span class="p">,</span> <span class="ss">polymorphic: </span><span class="kp">true</span><span class="p">,</span> <span class="ss">index: </span><span class="kp">true</span>
<span class="n">t</span><span class="p">.</span><span class="nf">date</span> <span class="ss">:valid_till</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h3>Set up Shrine and implement a behavior of the uploader</h3>
<p>After shrine has been installed and all needed plugins were added, we need to specify the storages.</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># config/initializers/shrine.rb</span>
<span class="k">def</span> <span class="nf">storage_cache</span>
<span class="no">Shrine</span><span class="o">::</span><span class="no">Storage</span><span class="o">::</span><span class="no">Sql</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">database: </span><span class="n">db_connect</span><span class="p">,</span> <span class="ss">table: :binary_files</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">storage_aws</span>
<span class="no">Shrine</span><span class="o">::</span><span class="no">Storage</span><span class="o">::</span><span class="no">S3</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="o">**</span><span class="n">s3_options</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div>
<p>Shrine uploads a file in two stages: <code>cache</code> and <code>store</code>. On the first stage by assigning a file to model it is being synchronously stored to DB ( We’ve chosen DB as cache because our app runs on few servers and we need one place for temporarily stored files ). And after validation and processing in our uploader, the second stage starts and the file is being promoted in background job to S3 storage.</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># app/uploader/our_uploader.rb</span>
<span class="k">class</span> <span class="nc">OurUploader</span> <span class="o">&lt;</span> <span class="no">Shrine</span>
<span class="n">plugin</span> <span class="ss">:backgrounding</span>
<span class="nb">self</span><span class="p">.</span><span class="nf">storages</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">cache: </span><span class="n">storage_cache</span><span class="p">,</span> <span class="ss">store: </span><span class="n">storage_store</span> <span class="p">}</span>
<span class="c1"># this job starts after file has been cached.</span>
<span class="no">Attacher</span><span class="p">.</span><span class="nf">promote</span> <span class="p">{</span> <span class="o">|</span><span class="n">data</span><span class="o">|</span> <span class="no">OurUploadingJob</span><span class="p">.</span><span class="nf">perform_later</span><span class="p">(</span><span class="n">data</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div>
<h3>Add Job to upload files in background</h3>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># app/jobs/our_uploading_job.rb</span>
<span class="k">class</span> <span class="nc">OurUploadingJob</span> <span class="o">&lt;</span> <span class="no">ActiveJob</span><span class="o">::</span><span class="no">Base</span>
<span class="k">def</span> <span class="nf">perform</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="n">attacher</span> <span class="o">=</span> <span class="no">OurUploader</span><span class="o">::</span><span class="no">Attacher</span><span class="p">.</span><span class="nf">load</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="n">record</span> <span class="o">=</span> <span class="n">attacher</span><span class="p">.</span><span class="nf">record</span>
<span class="n">cached</span> <span class="o">=</span> <span class="n">attacher</span><span class="p">.</span><span class="nf">get</span>
<span class="n">pdf_id</span> <span class="o">=</span> <span class="n">cached</span><span class="p">.</span><span class="nf">id</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">).</span><span class="nf">first</span><span class="p">.</span><span class="nf">to_i</span>
<span class="n">binary</span> <span class="o">=</span> <span class="no">BinaryFile</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="n">pdf_id</span><span class="p">)</span>
<span class="n">record</span><span class="p">.</span><span class="nf">binary_file</span> <span class="o">=</span> <span class="n">binary</span> <span class="c1"># Here we bind current record with 'binary_files' to have a fallback mechanism.</span>
<span class="n">record</span><span class="p">.</span><span class="nf">save</span>
<span class="n">promoted</span> <span class="o">=</span> <span class="n">attacher</span><span class="p">.</span><span class="nf">promote</span><span class="p">(</span><span class="n">cached</span><span class="p">,</span> <span class="ss">action: :store</span><span class="p">)</span>
<span class="k">if</span> <span class="n">promoted</span>
<span class="n">record</span><span class="p">.</span><span class="nf">update_column</span><span class="p">(</span><span class="ss">:uploaded</span><span class="p">,</span> <span class="kp">true</span><span class="p">)</span> <span class="c1"># Set 'uploaded' as true if file was successfully uploaded.</span>
<span class="n">binary</span><span class="p">.</span><span class="nf">update_column</span><span class="p">(</span><span class="ss">:valid_till</span><span class="p">,</span> <span class="mi">30</span><span class="p">.</span><span class="nf">days</span><span class="p">.</span><span class="nf">from_now</span><span class="p">)</span> <span class="c1"># Binary file will be removed from db after 30 days of uploading.</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h3>Add file retrieving service</h3>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># app/services/some_get_attachment.rb</span>
<span class="k">class</span> <span class="nc">SomeGetAttachment</span> <span class="o">&lt;</span> <span class="no">ActiveInteraction</span><span class="o">::</span><span class="no">Base</span>
<span class="k">def</span> <span class="nf">execute</span>
<span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">uploaded?</span>
<span class="n">fetch_from_s3</span>
<span class="k">else</span>
<span class="n">fetch_from_fallback_storage</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">fetch_from_fallback_storage</span>
<span class="k">if</span> <span class="n">document</span><span class="p">.</span><span class="nf">binary_file</span>
<span class="n">fetch_from_binary</span>
<span class="k">elsif</span> <span class="n">attacher</span><span class="p">.</span><span class="nf">cached?</span>
<span class="n">fetch_from_cache</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h3>Write the Attacher module</h3>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># app/models/concerns/our_attacher</span>
<span class="k">class</span> <span class="nc">OurAttacher</span> <span class="o">&lt;</span> <span class="no">Module</span>
<span class="k">module</span> <span class="nn">InstanceMethods</span>
<span class="c1"># File retriving service provides link on uploaded file.</span>
<span class="k">def</span> <span class="nf">readable_url</span>
<span class="c1"># Ask SomeGetAttachment about file</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">downloadable_url</span>
<span class="c1"># Ask SomeGetAttachment about file</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="c1"># Here we use module_eval method to have the ability to pass any attribute name when we include the uploader</span>
<span class="c1"># and also add the association with `binary_files` to the model.</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">column_name</span><span class="p">)</span>
<span class="vi">@column_name</span> <span class="o">=</span> <span class="n">column_name</span>
<span class="nb">module_eval</span> <span class="o">&lt;&lt;-</span><span class="no">RUBY</span><span class="p">,</span> <span class="kp">__FILE__</span><span class="p">,</span> <span class="kp">__LINE__</span> <span class="o">+</span> <span class="mi">1</span><span class="sh">
def self.included(model)
model.include InstanceMethods
model.include OurUploader[@column_name]
model.has_one :binary_file, as: :attachable
end
</span><span class="no"> RUBY</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>And include OurAttacher module to some model.</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># app/models/some_model.rb</span>
<span class="k">class</span> <span class="nc">SomeModel</span> <span class="o">&lt;</span> <span class="no">AcriveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="kp">include</span> <span class="no">OurAttacher</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:&lt;</span><span class="n">attribute_name</span><span class="o">&gt;</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div>
<h3>Conclusion</h3>
<p>As result, we have the stable web application with universal upload/retrieved mechanism that can be applied to any ActiveRecord model. This solution can use as an approach to any application thanks to Shrine and its modularity nature since it supplies a lot of plugins and allows to add plugins making by yourself, the process of uploading become much easier and flexible.</p>
Synchronization of multi threads between Sidekiq and PostgreSQLhttp://elixirator.com/blog/2017/synchronizations-multithreads-beetwen-sidekiq-and-postgresql.html2017-06-30T00:00:00+03:002017-08-01T17:09:05+03:00Vitalii Kozytskyi<p><img src="/blog/2017-06-30-synchronizations-multithreads-beetwen-sidekiq-and-postgresql/sidekiq-mailer.png" alt="image-title-here">
In our project, we are using Sidekiq as a background processing and PostgreSQL as a database. Our scheduler worker runs some task which sends emails for users and repeats each 15 seconds. Sometimes, in a case when we have many big tasks, that spend more than 15 seconds there are some conflicts between different threads in the Sequel.
</p><p><img src="/blog/2017-06-30-synchronizations-multithreads-beetwen-sidekiq-and-postgresql/sidekiq-mailer.png" alt="image-title-here" />
In our project, we are using Sidekiq as a background processing and PostgreSQL as a database. Our scheduler worker runs some task which sends emails for users and repeats each 15 seconds. Sometimes, in a case when we have many big tasks, that spend more than 15 seconds there are some conflicts between different threads in the Sequel.
</p>
<p>Imagine that our sidekiq is busy for 5 minutes, during this time 5*4=20 tasks of mails were created. At some moment 5 big tasks got finished and we have empty 5 threads concurrency. This means at that moment we started 5 mails tasks. In our code of worker, we have some <code>select</code> from the database. And after we work with that data and run personal worker for each mail. But PostgreSQL allows to use many selects in one moment and lock our table when we change some data. Imagine, that we have one mail which is ready to deliver. So we got 1 mail in one moment 5 times, for 5 parallel threads concurrent and are going working with them. Certainly, we update our data but we do this after select, and, actually, we update one entry 5 times. As a result, we got a mail from the database and create 5 the same workers for them. After that we started running personal worker for mail, if it was successful we’ll change the state of mail. But usually each mail sending time is nearly 500 milliseconds, and this is too much time for the server. So in one moment, we have in order 5 same mails with same data that are pending to be delivered. As a result, we will send 5 times one mail and our user will get 5 clone-mails.</p>
<h3>Fix with unique jobs:</h3>
<p>Firstly we thought: ‘Why do we create so many workers?”. We can create only one worker and wait until it will be finished. This is a good solution and we are using this way to work. After some time we’ve found some gem that does that instead of us. It&rsquo;s called <code>sidekiq-unique-jobs</code>. That&rsquo;s a good gem which in <code>sidekiq_options</code> has different attributes and checks the unique index of the worker where this attribute was added. Actually, we are using:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="n">sidekiq_options</span> <span class="ss">unique: :until_executed</span>
</code></pre></div>
<p>This is a normal worker and can be tested but imagine that we have many servers and each one of them runs mails schedule. And we don’t have the same workers at the same moment of time on the one server but we can have the same worker on other server and do the same actions concurrently. So the problem with reading the data is remaining. And we’ve started looking for a decision to fix this trouble by PostgreSQL again.</p>
<h3>Lock in PostgreSQL</h3>
<p>In PostgreSQL, we have some great method called “lock”. So what this method is doing? This method locks all table for actions until they would be finished. The basic syntax looks like:</p>
<div class="highlight"><pre class="highlight sql"><code><span class="k">LOCK</span> <span class="k">TABLE</span> <span class="n">mails</span> <span class="k">IN</span> <span class="k">EXCLUSIVE</span> <span class="k">MODE</span><span class="p">;</span>
</code></pre></div>
<p>But, I must emphasize, it does not enough. Because PostgreSQL locks the table only in some transaction. So every lock must be in the transaction, and we can write the next base:</p>
<div class="highlight"><pre class="highlight sql"><code><span class="k">BEGIN</span> <span class="k">WORK</span><span class="p">;</span>
<span class="k">LOCK</span> <span class="k">TABLE</span> <span class="n">mails</span> <span class="k">IN</span> <span class="k">EXCLUSIVE</span> <span class="k">MODE</span><span class="p">;</span>
<span class="o">***</span><span class="n">doing</span> <span class="k">some</span> <span class="n">actions</span> <span class="k">with</span> <span class="n">mails</span><span class="o">***</span>
<span class="k">COMMIT</span> <span class="k">WORK</span><span class="p">;</span>
</code></pre></div>
<p>When we give an attention to method lock we can notice some mode. What does it mean? Actually, it’s our limitation. Because PostgreSQL has 8 different modes such as: <code>ACCESS SHARE, ROW SHARE, ROW EXCLUSIVE, SHARE UPDATE EXCLUSIVE, SHARE, SHARE ROW EXCLUSIVE, EXCLUSIVE, ACCESS EXCLUSIVE</code>. And each of them locks the table for some actions. For example, the ACCESS SHARE command acquires a lock of this mode on referenced tables. In general, any query that only reads a table and does not modify it will acquire this lock mode. EXCLUSIVE allows only concurrent ACCESS SHARE locks, i.e., it only reads from the table that can proceed in parallel with a transaction holding this lock mode. So you can choose the mode which is better for you. When we speak about locking we must remember about <code>pg_locks</code>. It’s a special view in PostgreSQL which shows what action is locking at the moment. And when you use your background worker which saves data of scheduling in PostgreSQL you can use this view to synchronize worker running. This view has many functions which can get different information about the order at some moment. But this functions only par information about some special lock type with a relation or something else which is included in that view. Our sidekiq works with redis and this is the main reason why we cannot use that functionality for synchronization.</p>
<p>So we have described the good opportunity for the locking, but what about the downside. I can show the code in ruby using Sequel:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="no">DB</span><span class="p">.</span><span class="nf">transactions</span> <span class="k">do</span>
<span class="no">Models</span><span class="o">::</span><span class="no">Mail</span><span class="p">.</span><span class="nf">lock</span><span class="p">(</span><span class="err">“</span><span class="no">EXCLUSIVE</span><span class="err">”</span><span class="p">)</span> <span class="p">{</span> <span class="nb">sleep</span><span class="p">(</span><span class="mi">120</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div>
<p>Imagine that we are doing some actions for 120 seconds. And it doesn’t matter how many different parallel workers run concurrently. They will wait in line. And only one process will work with data in the mails table. Each user who creates new email has to wait for the finish of this worker. That&rsquo;s because we blocked all actions with this table for 120 seconds. This is the bad side of the locking. Certainly, you can choose a different mode that only creates or updates data. And it&rsquo;s better to take the mode which has less locking for action that is used by other processes.</p>
<h3>Have we fixed this problem?</h3>
<p>In our code, we have some select with the limit method and after we update some data in the table. This is why we use two queries. Yes! Two queries it’s a bad solution, but in many times this works quicker rather than one query with many joins and distinct. Because when you have millions of data in some tables and to join them with similar data it takes more time rather than when you use 2 queries. And sometimes, this solution optimizes your code tenfold. But this is a topic for another article. So in our code, we returned some id after actions and then we create new jobs. We have decided to create one big query to update all data which includes another query. Because it locks our table for less time.</p>
<h3>Conclusion</h3>
<p>You can use PostgreSQL as synchronizer between the concurrent jobs. If you are using many queries in background worker, you must be sure that your worker is unique. In other way lock the table or create one big query to be ensured that your jobs are working with correct data.</p>
Multitenancy with Hanami and Sequelhttp://elixirator.com/blog/2017/multitenancy-with-hanami-and-sequel.html2017-06-09T00:00:00+03:002017-08-01T17:09:05+03:00Denis Kondratenko<p>Hi folks! Today I’d like to share the story how we built a multi-tenant app with <a href="hanamirb.org">Hanami framework</a>. What we were trying to achieve is to have a single application instance, but to separate the data per each country we were hosted in. For example, to run a single server with an app that would serve requests from <code>https://app.uk</code>, <code>https://app.fr</code> &amp; <code>https://app.nl</code>. Let me say in advance that we made it working :)
</p><p>Hi folks! Today I&rsquo;d like to share the story how we built a multi-tenant app with <a href="hanamirb.org">Hanami framework</a>. What we were trying to achieve is to have a single application instance, but to separate the data per each country we were hosted in. For example, to run a single server with an app that would serve requests from <code>https://app.uk</code>, <code>https://app.fr</code> &amp; <code>https://app.nl</code>. Let me say in advance that we made it working :)
</p>
<h3>How it all began</h3>
<p>You may be wondering why we chose <a href="http://sequel.jeremyevans.net">Sequel</a> if Hanami has <a href="https://github.com/hanami/model">Hanami::Model</a>? I&rsquo;d like to share our consideration behind this decision, so I&rsquo;ll begin with a little history retrospective.</p>
<h5>Hanami::Model story</h5>
<p>When we decided to migrate from our Rails app, Hanami was at the 0.8.0 version and hanami-model at 0.6.1. We were excited about repository pattern and wanted to use it, but it turned out that hanami-model had a global config, including DB URI, preventing us to switch DBs at the run-time. And that was our goal - to have a single instance of our app that would <em>talk</em> to different DBs depending on the runtime context. However, we have a decent alternative - <a href="rom-rb.org">ROM</a>, or the <em>Ruby Object Mapper</em> (version 2.0.1 at that time).</p>
<h5>ROM story</h5>
<p>Thanks to the design of <code>ROM::Repository</code>, we could pass configuration on its initialization. First, we registered all countries-specific configuration within the app container. Then, we created our wrapper around <code>ROM::Repository</code> and redefined its initialize method so whenever we instantiate our repo we get the right config. It worked pretty well and we switched to ROM. Yay! We were even more excited because ROM seemed much more advanced than what Hanami::Model could provide. We started to play with ROM and the more things we tried to implement, the more often we faced discrepancy with its documentation. Once we literally copy-pasted the code from the documentation and it didn&rsquo;t work. We had to open its source and figure out what was going on pretty often. That slowed down the whole team significantly - enough to search for other alternatives&hellip;</p>
<h5>Sequel story</h5>
<p>Sadly, but we haven&rsquo;t found other decent ORMs designed with repository pattern in mind. We weren&rsquo;t happy about that but had to switch back to Active Record. Hold on, it&rsquo;s not what you might think! I am talking about the <em>pattern</em> here. Of course, we didn&rsquo;t want to use ActiveRecord with Hanami. I would not forgive myself, at least. So we chose the Sequel and I am quite happy about that! ヽ༼ຈل͜ຈ༽ﾉ</p>
<p>Sequel is a mature ORM with lots of useful plugins and extensions. It is powerful, flexible, fast and has an awesome maintainer! BTW, rom-sql (adapter of ROM to work with relational databases) is using Sequel under the hood. It is flexible enough to allow us to define our own repositories and entities on top of it (and we already have a prototype of it, so stay tuned).</p>
<h4>Enough talking, gimme the code!</h4>
<p>Among many extensions, there are two related to multitenancy: <code>server_block</code> and <code>sharding</code>. We have to enable them first. Then we establish Sequel connection with the additional server configurations on application load. It looks like:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="no">DB</span> <span class="o">=</span> <span class="no">Sequel</span><span class="p">.</span><span class="nf">connect</span><span class="p">(</span><span class="s2">"postgres://default_db_url"</span><span class="p">,</span> <span class="ss">servers: </span><span class="vi">@servers</span><span class="p">)</span>
</code></pre></div>
<p>The first argument is a URL to the default database. Sequel needs it to map tables into models (and probably some other stuff). It may be with some data, but our is empty.</p>
<p><code>@servers</code> holds configuration for our databases and looks like:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="vi">@servers</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">fr: </span><span class="p">{</span> <span class="ss">database: </span><span class="s2">"postgres://fr_url"</span> <span class="p">},</span> <span class="ss">uk: </span><span class="p">{</span> <span class="ss">database: </span><span class="s2">"postgres://uk_url"</span> <span class="p">}</span> <span class="p">}</span>
</code></pre></div>
<p>In order to perform an operation on other than default database, we have to use <em>server_block</em> extension:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="no">DB</span><span class="p">.</span><span class="nf">with_server</span><span class="p">(</span><span class="ss">:uk</span><span class="p">)</span> <span class="k">do</span>
<span class="no">Message</span><span class="p">.</span><span class="nf">create</span><span class="p">(</span><span class="ss">body: </span><span class="s1">'Such wow'</span><span class="p">,</span> <span class="ss">author_id: </span><span class="mi">1</span><span class="p">,</span> <span class="ss">topic_id: </span><span class="mi">1</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div>
<p>OK, that&rsquo;s pretty awesome, but how do we know when should we write to <code>fr</code> or <code>uk</code> servers? Here comes the rack middleware! We simply have to fetch the domain from the incoming request and call next middleware within <code>with_server</code> block, like this:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="k">module</span> <span class="nn">Middleware</span>
<span class="k">class</span> <span class="nc">SwitchDatabase</span>
<span class="k">class</span> <span class="nc">CountryNotDefinedError</span> <span class="o">&lt;</span> <span class="no">StandardError</span><span class="p">;</span> <span class="k">end</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="vi">@app</span> <span class="o">=</span> <span class="n">app</span>
<span class="vi">@db</span> <span class="o">=</span> <span class="no">App</span><span class="o">::</span><span class="no">Container</span><span class="p">[</span><span class="s1">'config.db_connection'</span><span class="p">].</span><span class="nf">db</span>
<span class="vi">@country_switch</span> <span class="o">=</span> <span class="no">App</span><span class="o">::</span><span class="no">Container</span><span class="p">[</span><span class="s1">'config.country'</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">request</span> <span class="o">=</span> <span class="no">Rack</span><span class="o">::</span><span class="no">Request</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">tld</span> <span class="o">=</span> <span class="n">request</span><span class="p">.</span><span class="nf">host</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s1">'.'</span><span class="p">).</span><span class="nf">last</span>
<span class="n">tld</span> <span class="o">=</span> <span class="no">ENV</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="s1">'FORCE_DB'</span><span class="p">)</span> <span class="k">if</span> <span class="no">ENV</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="s1">'FORCE_DB'</span><span class="p">,</span> <span class="kp">nil</span><span class="p">)</span>
<span class="k">raise</span> <span class="no">CountryNotDefinedError</span> <span class="k">if</span> <span class="n">tld</span><span class="p">.</span><span class="nf">nil?</span> <span class="o">||</span> <span class="n">tld</span><span class="p">.</span><span class="nf">empty?</span>
<span class="vi">@country_switch</span><span class="p">.</span><span class="nf">switch</span><span class="p">(</span><span class="n">tld</span><span class="p">)</span>
<span class="vi">@db</span><span class="p">.</span><span class="nf">with_server</span><span class="p">(</span><span class="n">tld</span><span class="p">.</span><span class="nf">to_sym</span><span class="p">)</span> <span class="k">do</span>
<span class="vi">@app</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">ensure</span>
<span class="vi">@country_switch</span><span class="p">.</span><span class="nf">reset!</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<p>And include our middleware before hanami in <code>config.ru</code> file:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="n">use</span> <span class="no">Middleware</span><span class="o">::</span><span class="no">SwitchDatabase</span>
<span class="n">run</span> <span class="no">Hanami</span><span class="p">.</span><span class="nf">app</span>
</code></pre></div>
<p>Our application now switches to correct databases depending on the top-level domain. It is thread-safe, easy to adjust and everything is pretty explicit.</p>
<p>Voila (ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ </p>
<h3>Instead of the summary</h3>
<p>I&rsquo;ll add few more notes. <code>@country_switch</code> from the code above is an instance of a special class that represents the current country. It is a thread-safe and allows us to use sharding with background job processors like <a href="http://sidekiq.org">Sidekiq</a>. It is also easy to simulate specific country in specs and test bits of country-specific logic. I&rsquo;ll probably write about those things in the next articles but now I have to end.</p>
<p>Thanks and see ya!</p>
Using Hanami with Webpackhttp://elixirator.com/blog/2017/using-hanami-with-webpack.html2017-06-01T00:00:00+03:002017-07-07T10:43:51+03:00Vladimir Dralo<p>Here, at Elixirator, we had a major rework for one of our projects. We were moving project’s backend from Rails to <a href="http://hanamirb.org/">Hanami</a> as part of this task. The Rails application had a mix of the javascript code, including vanilla JS, Coffeescript, JQuery and React. The final goal was to get rid of all extra dependencies, and use only one approach for writing frontend code - React.js.
</p><p>Here, at Elixirator, we had a major rework for one of our projects. We were moving project&rsquo;s backend from Rails to <a href="http://hanamirb.org/">Hanami</a> as part of this task. The Rails application had a mix of the javascript code, including vanilla JS, Coffeescript, JQuery and React. The final goal was to get rid of all extra dependencies, and use only one approach for writing frontend code - React.js.
</p>
<p>We have decided to use <a href="https://webpack.github.io/">Webpack</a> as the main source of all frontend code, including Javascript and CSS. One of the first steps of the migration was making Hanami and Webpack work together.</p>
<p>After some initial testing, the best approach was found for our needs. Here is a short overview how it can be done.</p>
<h3>Webpack configuration</h3>
<p>First what is needed to be done, is Webpack. Hopefully, there is nothing difficult here. Just some basic Webpack stuff. Dev server needs to be started at some address, serving assets from the specific location.</p>
<p>An example of simple configuration could look like this:</p>
<div class="highlight"><pre class="highlight javascript"><code><span class="c1">// webpack.config.js</span>
<span class="p">{</span>
<span class="nl">output</span><span class="p">:</span> <span class="p">{</span>
<span class="nl">path</span><span class="p">:</span> <span class="nx">path</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="nx">__dirname</span><span class="p">,</span> <span class="s1">'..'</span><span class="p">,</span> <span class="s1">'public'</span><span class="p">,</span> <span class="s1">'assets'</span><span class="p">),</span>
<span class="nx">filename</span><span class="p">:</span> <span class="s1">'[name]-bundle.js'</span><span class="p">,</span>
<span class="nx">publicPath</span><span class="p">:</span> <span class="s1">'http://localhost:8080/assets/'</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div>
<h3>Hanami configuration</h3>
<p>Hanami configuration is easy thanks to built-in helpers, which allow consuming assets from CDN. It is needed to specify content security policy to accept Javascript, CSS, images, etc. from location different than a website address. Providing config with the same host and port (as in the Webpack config) allows achieving this.</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># application.rb</span>
<span class="n">configure</span> <span class="ss">:development</span> <span class="k">do</span>
<span class="n">security</span><span class="p">.</span><span class="nf">content_security_policy</span> <span class="sx">%(
script-src 'unsafe-eval' *;
connect-src 'self' *;
img-src 'self' data: *;
style-src 'unsafe-inline' *;
font-src 'self' *;
)</span>
<span class="n">assets</span> <span class="k">do</span>
<span class="n">compile</span> <span class="kp">false</span>
<span class="n">cdn</span> <span class="kp">true</span>
<span class="n">host</span> <span class="s1">'0.0.0.0'</span>
<span class="n">port</span> <span class="mi">8080</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h3>Going live</h3>
<p>Despite using React.js as our main frontend library, most of the views are still written in plain HTML. These views are using Hanami asset helpers, such as <code>image</code> and <code>javascript</code>. We wanted to keep them. And we wanted to keep digested assets provided to us by Hanami.</p>
<p>When I was looking for the solution, I have tried to solve manifest file generation from the Webpack side initially. But existing plugins were producing it with the format that was not understood by Hanami. After spending some time, trying to export manifest with JS, I realized that I can achieve same easily using plain Ruby.</p>
<p>Webpack already outputs filenames with digest hash inside to the specified folder. So all I need is just get all files in assets folder, and save them to the JSON in the format that could be understood by Hanami.</p>
<p>The final solution is really straightforward:</p>
<div class="highlight"><pre class="highlight ruby"><code><span class="c1"># assets_tasks.rake</span>
<span class="n">namespace</span> <span class="ss">:assets</span> <span class="k">do</span>
<span class="n">desc</span> <span class="s1">'Generate assets from webpack'</span>
<span class="n">task</span> <span class="ss">:generate</span> <span class="k">do</span>
<span class="n">pty</span> <span class="s1">'npm run build:development'</span>
<span class="n">assets_hash</span> <span class="o">=</span> <span class="p">{}</span>
<span class="no">Dir</span><span class="p">.</span><span class="nf">glob</span><span class="p">(</span><span class="s1">'./public/assets/**/*.*'</span><span class="p">).</span><span class="nf">each</span> <span class="k">do</span> <span class="o">|</span><span class="n">f</span><span class="o">|</span>
<span class="n">file</span> <span class="o">=</span> <span class="no">Pathname</span><span class="p">(</span><span class="n">f</span><span class="p">.</span><span class="nf">sub</span><span class="p">(</span><span class="s1">'./public'</span><span class="p">,</span> <span class="s1">''</span><span class="p">))</span>
<span class="n">folder</span> <span class="o">=</span> <span class="n">file</span><span class="p">.</span><span class="nf">dirname</span>
<span class="n">asset_name</span> <span class="o">=</span> <span class="n">file</span><span class="p">.</span><span class="nf">basename</span><span class="p">.</span><span class="nf">to_s</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s1">'_'</span><span class="p">).</span><span class="nf">first</span>
<span class="n">extension</span> <span class="o">=</span> <span class="n">file</span><span class="p">.</span><span class="nf">extname</span>
<span class="n">assets_hash</span><span class="p">[</span><span class="s2">"</span><span class="si">#{</span><span class="n">folder</span><span class="si">}</span><span class="s2">/</span><span class="si">#{</span><span class="n">asset_name</span><span class="si">}#{</span><span class="n">extension</span><span class="si">}</span><span class="s2">"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span> <span class="ss">target: </span><span class="n">file</span><span class="p">,</span> <span class="ss">sri: </span><span class="s1">''</span> <span class="p">}</span>
<span class="k">end</span>
<span class="no">File</span><span class="p">.</span><span class="nf">open</span><span class="p">(</span><span class="s1">'./public/assets.json'</span><span class="p">,</span> <span class="s1">'w'</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">f</span><span class="o">|</span> <span class="n">f</span><span class="p">.</span><span class="nf">write</span><span class="p">(</span><span class="n">assets_hash</span><span class="p">.</span><span class="nf">to_json</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre></div>
<h3>Summary</h3>
<p>As the result, we started to use the combination of Hanami and React.js as our main tools. Adding minor features like auto mounting of React components, using mini SPAs inside the application, made our experience really smooth and pleasant. Webpack integration was just one of the first steps of our journey from Rails to Hanami. But this is a really long story. Stay tuned.</p>