Caktus Bloghttp://www.caktusgroup.com/blog/Blog | Django Web Development | Raleigh Durham Chapel Hill | Caktus Consulting Groupen-usMon, 14 Jul 2014 17:28:13 -0400July 2014 ShipIt Day Recaphttp://www.caktusgroup.com/blog/2014/07/14/july-2014-shipit-day/This past Friday we celebrated another ShipIt day at Caktus. There was a lot of open source contribution, exploring, and learning happening in the office. The projects ranged from native mobile Firefox OS apps, to development on our automated server provisioning templates via Salt, to front-end apps aimed at using web technology to create interfaces...Mon, 14 Jul 2014 17:28:13 -0400http://www.caktusgroup.com/blog/2014/07/14/july-2014-shipit-day/<p>This past Friday we celebrated another ShipIt day at Caktus. There was a lot of open source contribution, exploring, and learning happening in the office. The projects ranged from native mobile Firefox OS apps, to development on our automated server provisioning templates via Salt, to front-end apps aimed at using web technology to create interfaces where composing new music or <a href="http://www.youtube.com/watch?v=HzS-7-bT9g4">performing Frozen’s Let It Go is so easy anyone can do it</a>.</p>
<p>Here is a quick summary of the projects that folks worked on:</p>
<p>Calvin worked on updating our own minimal CMS component for editing content on a site, <a href="https://github.com/caktus/django-pagelets">django-pagelets</a>, to work nicely with Django 1.7. He also is interested in adding TinyMCE support and making it easy to upload images and reference them in the block. If you have any other suggestions for pagelets, get in touch with <a href="https://twitter.com/ironfroggy">Calvin</a>.</p>
<p><img alt="ShipIt Day Project: Anglo-Saxon / French Etymology Analyzer" src="https://farm3.staticflickr.com/2935/14632922756_bcb3605c6f.jpg" class="img-responsive" /></p>
<p>Philip worked on a code to tag words in a text with basic information about their etymologies. He was interested in exploring words with <a href="http://en.wikipedia.org/wiki/List_of_English_words_with_dual_French_and_Anglo-Saxon_variations">dual French and Anglo-Saxon variations</a> eg “Uncouth” and “Rude”. These words have evolved from different origins to have similar meanings in modern English and it turns out that people often perceive the French or Latin derived word, in general, to be more erudite (“erudite” from Latin) than the Anglo-Saxon variant. To explore this concept, Philip harvested word etymologies from the XML version of the Wiktionary database and categorized words from in Lewis Carroll’s Alice In Wonderland as well as reports from the National Hurricane Center. His initial results showed that Carroll’s British background was evident in his use of language, and Philip is excited to take what he developed in ShipIt day and continue to work on the project.</p>
<p>Mark created a Firefox OS app, <a href="http://caktus.github.io/costanza/">Costanza</a>, based on a concept from a <a href="http://en.wikipedia.org/wiki/The_Soul_Mate">Seinfeld episode</a>. Mark’s app used standard web tools including HTML, CSS, and Javascript to build an offline app that recorded and played back audio. Mark learned a lot about building apps with the new OS and especially spent a lot of time diving into issues with packaging up apps for distribution.</p>
<p>Rebecca and Scott collaborated on work in porting an application to the latest and greatest Python 3. The migration of apps from Python 2 to Python 3 started off as a controversial subject in the Python community, but slowly there has been lots of progress. Caktus is embracing this transition and trying to get projects ported over when there is time. Rebecca and Scott both wrestled with some of the challenges faced with moving a big project on a legacy server over to a new Python version.</p>
<p>Dan also wrestled with the Python 2 to 3 growing pains, though less directly. He set out to create a reusable Django app that supported generic requirements he had encountered in a number of client apps while exporting data to comma separated value (CSV) format. But, while doing this, he ran into difference in the Python 2 and 3 standard libraries for handling CSVs. Dan created <a href="https://github.com/caktus/cordwainer">cordwainer</a>, a generic CSV library that works both in Python 2 and 3. </p>
<p><img alt="ShipIt Day Project: Template Include Visualization" src="https://farm4.staticflickr.com/3914/14655514102_57b1d35649.jpg" class="img-responsive" /></p>
<p>Victor and Caleb worked together to create a wonderful tool for debugging difficult and tangled Django template includes. The tool helps template developers edit templates without fear that they won’t know what pages on the live site may be affected by their changes. They used d3 to visualize the template in a way that was interactive and intuitive for template writers to get a handle on complex dependency trees.</p>
<p>Michael has been working on a migraine tracking app using in iOS using PhoneGap and JQuery mobile. He has been diving in and learning about distributing mobile apps using XCode and interfacing with the phone calendar to store migraine data. In terms of the interface, Michael studied up on accessibility in creating the app whose primary audience will not be wanting to dig into small details or stare at their bright phone long while enduring a migraine.</p>
<p>Karen, Vinod, and Tobias all worked together to help improve Caktus’ Django project template. Karen learned a lot about updating projects on servers provisioned with Salt while trying to close out <a href="https://github.com/caktus/django-project-template/issues/107">one of the tickets</a> on our project-template repository. The ticket she was working on was how to delete stale Python byte code (.pyc) files that are left over when a Python source code file (.py) is deleted from a Git repository. These stale .pyc files can cause errors when they aren’t deleted properly during an upgrade. Vinod <a href="https://github.com/caktus/django-project-template/pull/111">worked through many issues</a> getting Docker instead of Virtualbox with Vagrant to create virtual environments in which SaltStack can run and provisioning new servers. Docker is a lighter weight environment than a full Virtualbox Linux server and would allow for faster iteration while developing provisioning code with SaltStack. Tobias improved the default logging configuration in the template to make it easier to debug errors when they occur, and also got started on some tools for integration testing of the project template itself.</p>
<p>Wray and Hunter collaborated to build a music composition and performance app called <a href="http://huntermacd.github.io/whoppy">Whoppy</a> (go ahead and try it out!). Whoppy uses <a href="http://www.w3.org/TR/webaudio/">Web Audio</a> to create a new randomized virtual instruments every time you start the app. Wray and Hunter worked through creating a nice interface that highlights notes in the same key so that it is easier for amateur composers to have fun making music.</p>ShipIt Day 4: SaltStack, Front-end Exploration, and Django Corehttp://www.caktusgroup.com/blog/2013/10/29/shipit-day-4-saltstack-front-end-exploration-and-django-core/Last week everyone at Caktus stepped away from client work for a day and a half to focus on learning and experimenting. This was our fourth ShipIt day at Caktus, our first being almost exactly a year ago. Each time we all learn a ton, not only by diving head first into something new, but...Tue, 29 Oct 2013 12:51:54 -0400http://www.caktusgroup.com/blog/2013/10/29/shipit-day-4-saltstack-front-end-exploration-and-django-core/<p>Last week everyone at Caktus stepped away from client work for a day and a
half to focus on learning and experimenting. This was our fourth ShipIt day at
Caktus, our first being almost exactly a year ago. Each time we all learn a
ton, not only by diving head first into something new, but also by hearing the
experiences of everyone else on the team.</p>
<h2>DevOps: Provisioning with SaltStack &amp; LXC+Vagrant</h2>
<p>We have found SaltStack to be useful in provisioning servers. It is a Python
based tool for spinning up and configuring servers with all of the services
that are needed to run your web application. This work is a natural
progression from the previous work that we have done at Caktus in <a href="http://www.caktusgroup.com/blog/2010/04/22/basic-django-deployment-
with-virtualenv-fabric-pip-and-rsync/">deploying
code in a consistent way to
servers</a>. SaltStack really shines with larger
more complicated systems designed for high availability (HA) and scaling where
each service runs on its own server. Salt will make sure that the setup is
reproducible.</p>
<p>This is often important while testing ideas for HA and scaling. The typical
cycle looks like:</p>
<ul>
<li>
<p>Work on states in Salt </p>
</li>
<li>
<p>Run Salt through Fabric building the whole system, locally through vagrant, on a cloud provider, or on physical hardware. </p>
</li>
<li>
<p>Pound on the system using benchmarking tools narrowing in on bottlenecks and single points of failure. </p>
</li>
<li>
<p>Start fixing the problems you uncovered in your states starting the cycle over again.</p>
</li>
</ul>
<p>Victor, Vinod, David, and Dan all worked on learning more about SaltStack
through scratching different itches they’ve felt while working on client
projects. Some of the particular issues folks looked at included understanding
the differences between running Salt with and without a master, how to keep
passwords safe on the server while sharing them internally on the development
team, and setting up Halite, a new web interface for Salt.</p>
<p>In order to test these complicated server configurations, we often rely on
Vagrant to run these full system configurations on developer’s laptops. This
is the quickest way to start building out systems. The problem with this
though is that our laptops are not as fast as the hardware that the services
will eventually be provisioned on. In order to reduce the time waiting in the
code-rebuild-test cycle, Scott our friendly system administrator delved into
running Vagrant with LXC containers on the backend with our development
laptops. LXC containers are more lightweight than VirtualBox virtual machines
and can be created more quickly on the fly. This involved learning about how
to upgrade the development laptop’s kernels on Ubuntu long term support image.
Scott was successful and there was a response of “Oooh” and “Ahhh” from the
developers when he demoed the speed of creating a new VM with LXC through
vagrant.</p>
<h2>Front-end Web + Visualizations: Realtime Web, Cellular Automata, Javascript</h2>
<p>MVC, &amp; SVG Splines</p>
<p>Caktus has a growing team of front-end developers and folks interested in user
interaction. There were a number of projects this ShipIt day that explored
different tools for designing visualizations and user experiences.</p>
<p>Mark dove into WebRTC building a new project, <a href="https://github.com/caktus/rtc-demo">rtc-
demo</a> with the <a href="http://caktus.github.io/rtc-demo/">actual demo
</a>hosted on Github static pages (note: it
only works on recent Firefox builds so far). It’s neat that this is hosted on
a static web assets host since the project does allow users to interact with
one another. Red &amp; Blue users go to the static site and create links that they
share with one another and connect directly through their browsers without any
server in between. Both users see a tic-tac-toe board and can alternate taking
turns. The moves are relayed directly to their opponent. This exploration
allowed Mark to evaluate some new technologies and their support in different
browsers. He was able to play around with new modes of interaction on the web
where the traditional server-client model can be challenged and hybrid peer-
to-peer apps can be built.</p>
<p><img alt="" src="http://upload.wikimedia.org/wikipedia/commons/1/13/Wireworld_XOR-gate.gif" /></p>
<p>Caleb has been taking a class on programming in C outside of work and wanted
to continue to work on one of his personal projects during the ShipIt day.
There’s something about pointer arithmetic and syntax that Caleb finds
fascinating Caleb extended some of the features of his cellular automata
simulation engine, <a href="https://github.com/calebsmith/gameoflife">gameoflife</a>.
This included some of his rule files, initial states, and rule file engine.
The result was experiments with the
<a href="http://en.wikipedia.org/wiki/Wireworld">Wireworld</a> simulation including a
working XOR gate using simple cellular automata rules and <a href="http://en.wikipedia.org/wiki/Brian's_Brain">Brian’s
Brain</a> which produces a visually
interesting chaotic simulation. Caleb’s demo was a crowd pleaser with everyone
enthralled by the puffers, spaceships, and gliders moving around and
interacting on the screen.</p>
<p>Rebecca made some amazing usability improvements to our own <a href="https://github.com/caktus/django-timepiece">django-
timepiece</a>. Timepiece is part of
the suite of tools that we use internally for managing contracts, hours,
timesheets, and schedules. Rebecca focused on some of the timesheet related
workflows including verifying timesheets. She <a href="https://github.com/caktus/django-timepiece/blob/timesheet-
redesign/timepiece/static/timepiece/js/timesheet/app.js">made these processes more
fluid</a> by building an API
and making calls to it in using Backbone. This allowed users to do things like
delete or edit entries on their timesheet before verifying it without leaving
the page.</p>
<p>Wray was also decided to work on a front-end project. His project focused on
building SVGs on the fly in raw Javascript without any supporting libraries.
Wray started by showing us his passion for how to mathematically define curves
and how useful the different definitions are from a user experience point of
view as designer working in a vector drawing program. The easier it is for the
designer to get from what they meant to draw to what is represented on the
screen the better the resulting design will be. He showed a particular
interest in the <a href="http://www.levien.com/spiro/">Spiro curve definition</a>. Wray
dug into this further by looking at the definition of curves supported by the
SVG standard and built an interactive tool for drawing curves by editing the
string defining the SVG element on the fly. The resulting
<a href="https://github.com/wraybowling/curvemixer">project</a> is still experimental at
this point, but is an interesting exploration into what can be done with SVGs
in Javascript without any supporting libraries.</p>
<h2>Django Core</h2>
<p>Karen and Tobias waded into some internals of bleeding edge of Django. In
particular, Tobias offered up comments on a <a href="https://github.com/django/django/pull/1758">pull
request</a> related to ticket
<a href="https://code.djangoproject.com/ticket/21271">#21271</a>. Tobias gave feedback to
the original pull request author and Django Core committer <a href="https://github.com/timgraham">Tim
Graham</a> on the difference between instance and
class variables and when it’s appropriate to use class variables in Python.</p>
<p>Karen gave an illuminating talk on transitioning a particular client project
to Django 1.6 from Django 1.5. Her <a href="http://talks.caktusgroup.com/shipit/2013/django16-upgrade-
experience/#/">slides are
available</a> to view online. She generalized the testing, debugging, and
eventually fixing strategies she used while discussing the particular problems
she encountered on the project. The strategy Karen used was running
<a href="https://docs.djangoproject.com/en/dev/ref/django-
admin/#check"><code>manage.py check</code></a> to check for new settings that may have changed. This is a new
management comment added in 1.6 and can be used from now on to ease upgrading
versions.</p>
<p>She found the following issues when upgrading:</p>
<ul>
<li>
<p><code>UNUSABLE_PASSWORD</code> was part of an internal API that changed. We used this internal in the project as sometimes you must, but special care must be taken when upgrading code that relies on unstable APIs. </p>
</li>
<li>
<p><code>MAXIMUM_PASSWORD_LENGTH</code> was used on this project after the recent Django security fix. Based on subsequent security updates, this is no longer needed and can be removed in 1.6. </p>
</li>
<li>
<p><a href="https://docs.djangoproject.com/en/dev/releases/1.6/#booleanfield-no-longer-defaults-to-false"><code>BooleanFields</code> now do not automatically default to <code>False</code>. </a>This was changed to encourage coder to be explicit and to increase the level of standards within the code. </p>
</li>
<li>
<p>Django debug toolbar broke in 1.6! This is a shame, but will hopefully be updated by the time 1.6 comes out or soon after.</p>
</li>
</ul>
<p>After this, Karen ran the project’s test suite and discovered the following
remaining changes:</p>
<ul>
<li><a href="https://docs.djangoproject.com/en/dev/releases/1.6/#default-session-serialization-switched-to-json">The default session serializer used to be picked. Now it’s JSON.</a></li>
<li><a href="https://docs.djangoproject.com/en/dev/releases/1.6/#boundfield-label-tag-now-includes-the-form-s-label-suffix"><code>BoundField.label_tag</code> now includes the form’s label_suffix</a>.</li>
</ul>
<p>Karen urged us to check out the <a href="https://docs.djangoproject.com/en/dev/releases/1.6/#booleanfield-no-
longer-defaults-to-false">Django 1.6 release
notes</a> and remember where you’re using unsupported
internals and to check and write tests for that code particularly carefully.
Also, she encouraged everyone to run <code>python -Wall manage.py test</code> to help
expose more deprecation warnings and bugs.</p>
<h2>Estimation Process Review</h2>
<p>Ben and Daryl, our Project Managers at Caktus, worked up a full proposal and
project schedule for an internal process of reworking our estimation process.
We don’t shy away from large fixed bid projects and pride ourselves on
meticulous estimates informed by careful client communication and requirements
gathering. Our PMs wanted to help us look at this process and make it more
formal as we grow to a larger group with lots of people leading the same
process for different incoming projects.</p>
<h2>Wrap Up</h2>
<p>We had a great time with our latest ShipIt day. Each time we learn a ton from
each other and by digging into some tough technical problems. We also learn a
little bit more about what makes a good ShipIt day project. There were a
number of crowd pleasers in this batch of projects and while some folks
decided to try a completely new library or technology others decided to stay a
bit closer to home by extending learning they did not have time look into more
during a client project. We had a lot of fun and are looking forward to our
next ShipIt day.</p>