spbnick pokes techA blog about all things technology, but mostly computer software and hardware.
http://spbnick.github.io/
Thu, 07 Dec 2017 08:52:33 +0000Thu, 07 Dec 2017 08:52:33 +0000Jekyll v3.6.2DIGImend is dead, long live DIGImend!<p>So, the old <a href="http://digimend.github.io">DIGImend</a> is
<a href="http://spbnick.github.io/2016/07/31/Wrapping-up-DIGImend-work.html">dead</a>.
I’m no longer going to be doing much volunteer work on it, and although there
were several people approaching me with the interest of taking over, nobody
has really stepped up yet.</p>
<p>I thought about this situation for a while, feeling sad about all the unused
skills I acquired through these years working on the project, and pondered my
motivation. I couldn’t arrive at a solution. Then I got contacted by <a href="http://neoblast.co.jp/">Neoblast
Inc.</a> and <a href="http://xp-pen.com/">XP-Pen</a>, saying they
were alarmed by DIGImend shutting down and offering their financial support.
Neoblast Inc. is a Japanese company building affordable creativity-oriented
computers for moms and school PTAs, which run Linux and employ affordable
graphics tablets. XP-Pen is one of graphics tablet makers, which Neoblast Inc.
chose as a partner.</p>
<!--more-->
<p>I secured a permission to do paid work in my free time from my employer, and
after a while we worked out the financing terms. They sent me three tablets to
make drivers for, and I started working. The work done so far, implementing
support for two tablets, was streamed to YouTube:</p>
<iframe width="560" height="315" frameborder="0" gesture="media" allow="encrypted-media" src="https://www.youtube.com/embed/tXxNhw0TXSM?list=PL9pEpI5Ro3ie4QHYYh8ALq2Zi4iRV6GmM" allowfullscreen="">
</iframe>
<iframe width="560" height="315" frameborder="0" gesture="media" allow="encrypted-media" src="https://www.youtube.com/embed/TOufzf-U2Ko?list=PL9pEpI5Ro3ifwV7yGNE2kEQ4CpZSer96C" allowfullscreen="">
</iframe>
<p>That and seeing creative people on YouTube get funded through Patreon brought
me into thinking that I can get some funding too, rationalize such work to
myself and my family, get motivation, and perhaps even get funded enough to
take some time off work for DIGImend.</p>
<p>Then I got contacted by <a href="https://www.oeamtc.at/">ÖAMTC</a> - apparently a car
owner’s club or association (sorry, I don’t know German), working on a project
using a signature pad. They wanted Linux drivers. I agreed and made the first
version of the driver:</p>
<iframe width="560" height="315" gesture="media" allow="encrypted-media" src="https://www.youtube.com/embed/7nxsfTYl4AY?rel=0" frameborder="0" allowfullscreen="">
</iframe>
<p>At the same time, people started asking for drivers for the refreshed Huion
tablets. Apparently Huion switched to new hardware utilizing somewhat
different initialization and report protocols, incompatible with the old ones,
and they don’t work on Linux. Some people offered to buy me a beer or send
donations. This sealed my resolve and I finally brought my self up to setup
<a href="https://www.patreon.com/spbnick">my own Patreon page</a>. I’m still working out
rewards and goals, but this should be good for the start. I’ll be glad to hear
your comments and advice about it!</p>
<p>I hope, of course, to get enough funding to get some time off work and
dedicate it to DIGImend eventually, but before that happens I’ll do the work
proportional to the pledges I get, and do work on request from companies, in
my free time. I’ll be contacting tablet manufacturers, again, to try to
interest them in funding my work.</p>
<p>So, if you need driver for your tablet, or would just like to see more
affordable tablets supported by Linux, consider pledging your support here:</p>
<p><a href="https://www.patreon.com/bePatron?u=4996347" data-patreon-widget-type="become-patron-button">Become a Patron!</a>
<script async="" src="https://c6.patreon.com/becomePatronButton.bundle.js"></script></p>
<p>If you’re a company and would like specific drivers done for you, a part of
the Linux input stack improved, or bugs fixed, write to
<a href="mailto:spbnick@gmail.com">spbnick@gmail.com</a>.</p>
Sun, 03 Dec 2017 00:00:00 +0000http://spbnick.github.io/2017/12/03/Long-live-DIGImend.html
http://spbnick.github.io/2017/12/03/Long-live-DIGImend.htmlA gooseneck fan from an IKEA Jansjö<p><img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/jansjö_fan.jpg" alt="Jansjö fan" /></p>
<p>Everyone knows the <a href="http://www.ikea.com/us/en/catalog/products/20315674/">IKEA Jansjö lamp</a> - it’s small, light, and
flexible, both figuratively and literally. Just the right spot lamp for all
kinds of smaller work - clip it to your table, shelf, cabinet, whatever, then
bend and point as necessary. Its low price and widespread availability has
lent it to a plethora of hacks: <a href="http://revolutionary-turners.com/attach/RW_LED_Lamp.pdf">a magnet base hack</a>,
<a href="https://youtu.be/n6JkUO8AHOQ">another one</a>, and <a href="https://youtu.be/FPKWRcUgyBg">another</a>, and <a href="https://youtu.be/QaSXGQ3bhIo">still
another</a>, a <a href="http://www.instructables.com/id/Lighted-Sheet-Music-Stand/">lighted sheet music
stand</a>, a <a href="http://www.instructables.com/id/Kids-Travel-Globe-Night-Light/">globe night light</a>, a
<a href="http://www.instructables.com/id/Flamingo-Lamp/">flamingo-shaped lamp</a>, and a ton of others.</p>
<!--more-->
<p>I was practicing hot-air desoldering recently, and, although I did so on my
balcony, I still had to breathe in quite a bit of fumes produced by heated
PCBs and components (ah, the smell of popping electrolytics!). So I thought a
bit of extra air movement could get them away from me, but my soldering fume
extractor would have been useless beside an already powerful air stream from
the hot-air gun. That’s when I thought something like a smaller fan closer to
my face would be just about right, something like <a href="http://hackaday.com/2016/10/02/a-desk-lamp-solder-fume-extractor/">this one, attached to a
desk lamp</a>. I was already using a Jansjö while desoldering, we
had a couple extra from a flea market (who would even sell them off?), and I
thought putting a fan on one would be just the thing.</p>
<p>Yeah a black Jansjö would be better, but I had a white one, so here it is.
What follows is how I made it, and how you can make one. I wanted to be able
to put the lamp back together and reuse the fan, if my plan didn’t work, so I
proceeded carefully, trying to preserve everything. However, I’m sure there
are a thousand ways to do this, and it’s up to you how gentle/barbaric you’re
going to be with yours. This is just my way.</p>
<p><strong>DISCLAIMER</strong>: I’m in no way responsible for any damage caused to the lamp,
yourself, your dog, or your apartment, as a result of following my
instructions. This will also void the lamp’s warranty. Proceed at your own
risk!</p>
<p>You’ll need:</p>
<ul>
<li>An <a href="http://www.ikea.com/us/en/catalog/products/20315674/">IKEA Jansjö</a>, with a base of your choice. I.e. a clip like
I used, a floor stand, a table stand - whatever you like, the lamp fixtures
themselves seem to be the same, and that is what matters.</li>
<li>A fan. If you’re going to use my mounting method, then it shouldn’t be
thicker than 10mm, which is what I used. Mine had 60mm sides, but smaller or
larger would also work, as long as the goose neck can hold its weight. I
used a <a href="http://www.ebay.com/itm/171929487253">Y.S.TECH FD126010LB</a> (no affiliation with the seller). This one
is not cheap, but is supposed to last long, being brushless and seemingly
having ball bearings.</li>
<li>A power adapter with a rating corresponding to the fan you have. The one
Jansjö has, while being powerful enough, has voltage (4V) which is too low
for any but the smallest fans. It must output DC (direct current), as small
fans are usually DC-powered! Make sure the adapter’s claimed maximum current
is at least twice as much as the current consumed by the fan, as those are
never exact. I picked a 12V 500mA-rated power adapter from a flea market for
my 140mA-rated fan.</li>
<li>A basic multimeter to test polarity when connecting the power supply.</li>
<li>Soldering iron and soldering wire.</li>
<li>A piece of heat-shrink tubing or electrical tape for insulating soldered
wires.</li>
<li>A glue gun with a glue stick, or something else to fix the wire to the
assembly. Electrical tape will probably hold it for a while.</li>
<li>A piece of thin sheet metal, like tinplate, e.g. from a tin can.</li>
<li>Metal shears to cut the metal.</li>
<li>Pliers to hold and bend it.</li>
<li>A small diameter machine screw with a flat head and a corresponding nut.
I used an M3 with a nylon locking nut.</li>
<li>A drill, a 2mm drill bit, and another one, somewhat larger than the screw
diameter, but smaller than its head, so the screw can sink into the hole.
I used a 3.5mm one, IIRC.</li>
<li>Something thin and sharp to pull out the lens fixture, I used two pins.</li>
<li>Something a bit thicker, like two small flat-head screwdrivers to help them.</li>
</ul>
<p>That’s one hell of a list, but all I can say is improvise if you lack
something.</p>
<p>The lamp and the fan I started with:
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/lamp_and_fan_before.jpg" alt="Lamp and fan" /></p>
<p>First I removed the lens fixture. It’s hard to fit anything under it, so I
used two pins helping each other, and squeezing the metal skirt gently to get
under the fixture’s lip. Then I was able to put two small flat screwdrivers
there.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/under_lens_fixture.jpg" alt="Pins under lens fixture" /></p>
<p>I worked my way around the fixture levering it up until it popped out, freeing
the lens and revealing the reflector. You can see the tabs which were holding
the lens fixture, on the sides of the reflector.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/lens_off.jpg" alt="Lens off" /></p>
<p>Unscrew the reflector. It’s held by 2mm cross-head machine screws. Save them.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/reflector_out.jpg" alt="Reflector out" /></p>
<p>Removing the reflector reveals the board with the LED and a few components.
De-solder the power supply wires and take the board out. Better do it with
pincers, as the thermal paste underneath will stick to anything.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/board_out.jpg" alt="Board out" /></p>
<p>Here is our empty lamp. From this point on we will stop removing things from
this end and will be adding them. Note that some thermal paste is still there.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/empty_lamp.jpg" alt="Empty lamp" /></p>
<p>Next I made two angled braces (what’s the proper term?) for the fan, from the
thin springy steel I salvaged from an inkjet printer a while ago. This wasn’t
an exact job with the materials and tools I had. There was no measuring
involved and I mostly used my eye and trying to fit things together, but here
are a few tips. Drill 2mm holes for fixing the braces to the lamp base,
drill holes slightly larger than the screw to hold the fan, but smaller than
the screw’s head (I used an M3 screw and a 3.5mm drill bit). Position the
holes for the fan-holding screw somewhat lower than the ideal placement, so
that when you screw the assembly to the lamp base, the 2mm screws would pull
the fan in, push it against the lamp skirt, and hold it firmly in place. Also,
make sure that when the fan-holding screw is in, it still allows the smaller
lamp’s screws to be in place and turned. Here are the braces alongside a 2mm
screw from the lamp.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/fan_braces.jpg" alt="Fan braces" /></p>
<p>Don’t even think about holding the metal with your hands while drilling,
unless you want your fingers finely shredded! The drill bit will inevitably
lock up in the thin metal and it will start spinning with it like there’s no
tomorrow. Hold the metal with a tool, e.g. pliers!</p>
<p>Here are the braces on the fan for a test. Note that you’ll have to take them
off again before proceeding.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/fan_braces_on.jpg" alt="Fan braces on" /></p>
<p>Next I attached the first bracket before soldering the fan wires, but it’s
probably easier the other way around. Don’t tighten that screw yet!
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/bracket_in.jpg" alt="Bracket in" /></p>
<p>Don’t forget to put your heat-shrink tube on <strong>before</strong> soldering the wires.
I cut the fan wires a bit shorter before this, so there wouldn’t be any excess
when it’s fixed to the lamp.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/wires_soldered.jpg" alt="Wires soldered" /></p>
<p>Now for the tricky bit. Put the fan-holding screw through the still unattached
bracket and then through one of the fan corner screw holes. Then bend the
already attached bracket outwards, towards the lamp skirt, so that you can put
the screw (which is in the fan) through it. Then slide the other bracket in
too, but don’t screw it in. Now you should be able to move the fan-holding
screw back and forth and also wiggle the fan around a bit. This should give
you enough flexibility to put the nut on the fan-holding screw. That and lots
of patience. Believe me, you’ll need it.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/nut_on.jpg" alt="Nut on" /></p>
<p>I admit I should have made more photos of this process, but at the time I
wasn’t sure this is going to work, and it was difficult enough, so this will
have to be another case of “<a href="http://imgur.com/gallery/RadSf">how to draw an owl</a>” meme.</p>
<p>Once you have the nut on and tightened, screw the other bracket to the lamp
and tighten both of the remaining screws, so that the fan is held in the lamp
skirt firmly. Try not to overdo it, so if you use a softer metal for the
braces they are not ripped, or the fan plastic doesn’t break. BTW, this is why
we needed the flat-headed screw for holding the fan - to be able to put the
second lamp-fixing screw in. Sorry for the over-exposure here.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/tightening_second_brace.jpg" alt="Tightening second brace" /></p>
<p>Next thing, you might want to fix the fan wire down so it doesn’t get broken
from vibration, doesn’t obscure the air flow, and doesn’t get under the fan
blades. I used some hot glue, but you might get away with some strong sticky
tape. I also left some extra wire length on the fan, and oriented the wire
entrance away from the lamp, because I wanted to be able to reuse the fan.
The corner of the fan where wire enters is the weakest one, and I didn’t want
to attach that to the lamp. No dedicated picture here, sorry, but see the top
and the bottom ones.</p>
<p>Then replace the power adapter. Cut the lamp’s and power adapter’s cord at the
places of your choice and solder them together. Observe polarity! If you do it
wrong and you have a brushless fan, then its control electronics might not
work, or even (theoretically) burn out. A basic DC motor would work, but would
turn in the wrong direction, possibly resulting in a suboptimal air flow. Try
checking the fan frame for arrows showing the rotation and air flow directions
to make sure it’s right, or use your multimeter and check wire colors.
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/power_adapter.jpg" alt="Power adapter" /></p>
<p>Do not repeat my mistake and do cut off the lamp’s cord <strong>before</strong> soldering
and isolating the wires inside the lamp, so you can test which one is which
from the power adapter side.</p>
<p>Anyway, I’ll pull the “<a href="http://imgur.com/gallery/RadSf">draw an owl</a>” here again, skip the
power cord soldering and insulation, and will rush you to the final picture:
<img src="/assets/2016-10-15-Gooseneck-fan-from-Jansjö/complete.jpg" alt="Complete" /></p>
<p>I’m very pleased with the result. Although I still prefer to bend the goose
neck itself, and more so when the fan is running, it is sitting in the lamp
nice and tight and I can bend the lamp by pulling the fan as needed, without a
hint of slack, or fear of breaking it off. I considered putting a wire mesh on
both sides of the fan for safety at the cost of efficiency, but where is the
fun in that? Make your own choice here, though. Otherwise, the fan is quiet,
airflow is great, and I’ll be testing it with my desoldering process soon!</p>
<p>Even if you don’t do it exactly as I did (completely understandable), I hope
my HOWTO inspires you to do something like this yourself. Enjoy your hacking!</p>
Sat, 15 Oct 2016 00:00:00 +0000http://spbnick.github.io/2016/10/15/Gooseneck-fan-from-Jansj%C3%B6.html
http://spbnick.github.io/2016/10/15/Gooseneck-fan-from-Jansj%C3%B6.htmlWrapping up DIGImend work<p>In short: I’m wrapping up my involvement in <a href="http://digimend.github.io/">DIGImend project</a>
within two months.</p>
<p>It was long apparent, but now I would like to state that officially, and stop
misleading users and tablet owners: I no longer have time nor energy to do the
work required. I’m sorry for all the false hope I may have given people, and
the promises I’ve broken in the meantime.</p>
<p>Starting today, I’m stopping all research on specific tablet interfaces and
protocols required for implementing drivers. I.e. I’m not going to respond to
any diagnostics, or requests to make new tablets work. I’m not going to
support users, or investigate their problems either. However, I will still be
reviewing and accepting patches, including ones already submitted.</p>
<p>For the following two months, until the end of October, I’m focusing first on
syncing the <a href="https://github.com/DIGImend/digimend-kernel-drivers">DIGImend kernel drivers</a> with the
upstream and writing HOWTOs on how to diagnose problems with tablets and how
to reverse-engineer tablet protocols.</p>
<p>I will also be available for coaching and support of any able person willing
to take my place, as a top priority. Write to <a href="mailto:spbnick@gmail.com">me</a>, if you would like to
step in, know C well, have experience with kernel and system programming and
interest in reverse-engineering USB devices, plus you have the patience to
deal with users of widely-varied experience and background.</p>
<p>Lastly, if anybody wishes to employ me to continue working on DIGImend, I’m
open to the offers, and I will gladly continue working on the project,
provided I’m appropriately compensated.</p>
<p>After those two months, in November, I’m going to stop all work on DIGImend,
except, if a replacement maintainer isn’t found, I will still be reviewing and
accepting patches as time allows. Otherwise, I will provide no support or help
with any user issues. I will also shutdown the DIGImend maillists, preserving
the archives, if possible, unless the new maintainer (if any) wishes
otherwise. The GitHub issues will remain open for discussions, all the code
and the website will remain online.</p>
<h2 id="digimend-history">DIGImend history</h2>
<p>In August 2005 my future wife and her friend gave me a <a href="http://digimend.github.io/tablets/UC-Logic_WP8060U/">Genius MousePen
8x6</a> as a birthday present. It almost worked in Linux,
but not quite. After looking for a while I found an Aiptek X11 driver hacked
by Jan Horak (then a student at Brno University of Technology) to work with
Genius WizardPen tablets. The driver was named WizardPen and hosted at the
university’s web-server. It didn’t work that well with my tablet, so I started
hacking. Eventually we made it work with mine and a bunch of other tablets. I
also added a calibration utility. We supported it for a while on the forum Jan
setup, and did a couple releases.</p>
<p>The driver worked by ignoring the fact that the kernel didn’t understand these
tablets well and reported garbage to the user-space, and simply tried to make
sense of that. This approach was flawed, but effective, and to this day the
<a href="https://launchpad.net/wizardpen">WizardPen driver</a>, now supported by Ubuntu community, is a go-to
for desperate tablet owners, as it sometimes works when there’s no support
otherwise.</p>
<p>I thought that such approach is limited, and decided to instead work at the
root of the problem: the kernel, relying on the catch-all evdev X11 driver in
user-space. While investigating a bunch of tablets I found that most of the
time the problem was in how cheap tablets described themselves. They
misinterpreted the USB HID standard, apparently tweaking the tablets only so
that they would pass the Windows certification/tests and often relying on
their custom Windows drivers to make them actually work. Otherwise, their
protocols were very straightforward, and if I could fix their report
descriptors, they would just work.</p>
<p>So, in July 2008 I started my own project and named it DIGImend (after
“digitizer mending”, “digitizers” being the old name for graphics tablets). I
began with making a couple diagnostics tools and basic kernel and evdev
patches. In 2009 I started building an <a href="https://github.com/DIGImend/hidrd">editor tool</a> for HID report
descriptors - data structures provided by input devices to describe their
protocols. The plan was to make the kernel ignore the device’s descriptors and
feed it ones that I fixed instead. Eventually the tool started working, I
contributed requisite fixes to the kernel and started working on the tablets.</p>
<p>I didn’t have money to buy them, so I went around computer hardware shops in
my city (then Saint-Petersburg, Russia) asking for permission to check how
tablets on display worked, and surprisingly most of the shops were fine with
that. So I had a bunch of tablets reverse-engineered and supported that way.
Later, after moving to Finland, I bought some tablets with user’s help and had
Waltop send me two of theirs. I put up a wiki on the sourceforge.net hosting
and a couple people joined, helped me fill it in, and helped support the
users.</p>
<p>I contributed patches to the upstream kernel and also released patch packages,
followed by patched kernel packages for Ubuntu. At some point sourceforge.net
abandoned MediaWiki support, forcing people to maintain their own installs,
which didn’t work that well, so I reworked and moved the project website to
GitHub. After out-of-tree kernel HID drivers became possible, and report
descriptor replacement started working for them, I made the DIGImend kernel
drivers package, which has now acquired a healthy popularity.</p>
<p>However, in the past year or so I found myself occupied otherwise: I started
getting more involved with my hobby of embedded programming and electronics,
and then I had eight months of intensive Finnish courses, followed by
continued self-study. My daughter is also growing up, and I find myself
needing to spend more time with her. All-in-all, I no longer have time nor
energy required to keep up with the development in the Linux input stack, new
tablets being produced, and the multitude of user requests coming in. I failed
people expectations many times already, and now it’s time to stop.</p>
<h2 id="thanks">Thanks</h2>
<p>I would like to thank everyone who cooperated, and helped me with this
project. I’m trying to remember as many people as I can, but please forgive
me if I forgot you, you have my thanks too! So, in no particular order:</p>
<ul>
<li>Jan Horak, who started the WizardPen driver development and with whom I
worked to improve it;</li>
<li>Peter Hutterer, the maintainer and developer of the evdev X11 driver, a
Wacom driver developer (among other things), and a fellow Red Hat engineer,
who accepted my patches and helped me understand how things worked;</li>
<li>Jiří Kosina, the maintainer of the HID subsystem in the Linux kernel (among
other things), for reviewing my patches and explaining things to me;</li>
<li>Greg Kroah-Hartman, maintainer of the USB subsystem in the Linux kernel and
usbutils developer (among many other things), and simply a wonderful person,
for also reviewing my patches and helping me learn;</li>
<li>Jason Gerecke, a Wacom driver developer, for helping me understand things
about the Wacom driver and contributing to hidrd;</li>
<li>Ping Cheng, a Wacom driver developer, for helping me understand the
Wacom driver;</li>
<li>Benjamin Tissoires, an input stack developer, and a fellow engineer at Red
Hat, for helping me with the Wacom driver too, and contributing to the
tablet drivers, both upstream and in the DIGImend kernel drivers package;</li>
<li>Martin Rusko, who did initial reverse-engineering of Huion tablets, hacked
on the driver and tested it;</li>
<li>Yann Vernier, for contributing the Polostar tablet driver;</li>
<li>Waltop International Corporation, who sent me two tablets and provided some
information on them;</li>
<li>Masaru Hirano from Neoblast Inc., Japan for providing information on Ugee
tablets they sell;</li>
<li>Yiynova Europe BV, who did initial Yiynova tablet support hacking and
then testing;</li>
<li>Favux, for helping support users and contributing wiki contents and HOWTOs;</li>
<li>Viktoria S., for helping support users and contributing HOWTOs;</li>
<li>Alexandre Prokoudine, of libregraphicsworld.org for writing about the
project;</li>
<li>the incredible number of tablet owners, who meticulously collected
diagnostics, strived to learn Linux, and tested the drivers, for their
efforts, patience and gratitude;</li>
<li>other engineers who contributed small and sometimes not so small fixes to
all the project software and the website;</li>
<li>all the people who helped, and still help fellow tablet owners on the
maillist and in the GitHub issues;</li>
<li>all the shop assistants who let me play with the tablets in the stores.</li>
</ul>
Sun, 31 Jul 2016 00:00:00 +0000http://spbnick.github.io/2016/07/31/Wrapping-up-DIGImend-work.html
http://spbnick.github.io/2016/07/31/Wrapping-up-DIGImend-work.htmlMaking sense of libtevent<p>I’m working on integrating <a href="http://scribery.github.io/tlog/">tlog</a> with <a href="https://fedorahosted.org/sssd/">SSSD</a> and am finding a
specific part of SSSD hard to understand. Among other things, it uses the
<a href="https://tevent.samba.org/">tevent</a> library, made by Samba developers to solve particularly hairy
problems of asynchronous programming. While its file descriptor, signal, and
timer event handling is relatively straightforward, I found its
“<a href="https://tevent.samba.org/tevent_request.html">requests</a>” feature confusing and hard to reason about.</p>
<p>My guess is it uses some CS terminology I’m not familiar with (not having
finished my university studies), and without that background it hardly makes
sense to me. Anyway, following the old adage “if you want to learn something,
teach it”, I’ll try to explain what the terminology used by the “requests”
part of tevent means. It will definitely help me, and I’ll be glad if someone
else finds it useful.</p>
<p><strong>DISCLAIMER</strong>: This is just my opinion and my attempt to understand tevent
better. I’m only starting to make sense of it, and can make serious mistakes.
Don’t take the below as facts or the best approach at understanding tevent,
use your own judgement. For that matter, I’ll be glad to receive comments,
corrections and suggestions for better approaches.</p>
<p>Tevent represents the “event loop” approach to asynchronous programming: it
implements an “<a href="https://tevent.samba.org/tevent_context.html">event context</a>” - basically, a list of
event/callback pairs, functions to create and modify it, and a function
waiting for any of the events in the list, and invoking their corresponding
callbacks when they happen. An <a href="https://tevent.samba.org/tevent_events.html">event</a> can be a file descriptor
state change, a signal, a timer firing, or a synthetic “immediate” event.</p>
<p>That is all fine and dandy while you have just a few of those, plus basic
processing in between - you can have all the code spread over callbacks,
sharing a bit of global state. Yet, things get hairy when you need to e.g.
take an input from one socket, wait for some input from another, send the
result to a third, and track this process for tens of clients at the same
time. That needs a good deal of shared data and progress tracking, and rolling
custom code each time you implement something like that quickly gets boring.</p>
<p>This is where tevent “requests” come in (named “tevent_req” in the code). A
“request” is just an abstraction of a piece of work done (i.e. code executed)
on a piece of data. It can be any code and any data whatsoever, working for
whatever purpose. You can give it the data to work on, a function to call when
it’s done, and afterwards you can ask it for the results of its work. There
are more housekeeping operations and data available, but that’s basically it.</p>
<p>And here we come to the first confusion: the “request” name. It’s not actually
a request. There is no entity being requested anything. A more suitable name
would perhaps be simply “work” (“tevent_work” in the code).</p>
<p>The library provides the following major functions to deal with requests:</p>
<ul>
<li>tevent_req_create - allocate and initialize a request (normally done from
within the request code, talloc is used to destroy),</li>
<li>tevent_req_set_callback - set a function to call when request is done
(normally done from outside the request code, by the user),</li>
<li>tevent_req_done - mark request done, and call the callback specified with
tevent_req_set_callback (normally done from within the request code),</li>
</ul>
<p>So far so good. However, the “tevent_req_set_callback” operation doesn’t give
a clue as to what that callback does or when it is called, plus you can
actually specify other callbacks, which makes it more confusing. Sure, it is
the only callback set by the user normally, but that is not clear from the
name, nor even from the documentation. Perhaps this operation would have
been better named “set_done_fn”, or, in full, “tevent_work_set_done_fn”.
That would also match the pattern used to name other callback-modification
functions.</p>
<p>The library documentation describes how to implement a request. It says, you
need to define a function with “send” on the end of its name, which would
accept data to work on, create a request using “tevent_req_create”, attach the
data to it, and would start the actual work. You also need to define a
function with “recv” (meaning “receive”) on the end of its name, which would
extract the work results from the request.</p>
<p>This convention follows the idea of a work request being “sent” somewhere for
execution and results being “received”. However, nothing is actually being
sent to, or received from anywhere. This is particularly confusing in code
actually sending/receiving something, e.g. network servers and clients.</p>
<p>If the “request” was named “work” instead, then these two functions could be
named “begin”, and “reap”, or “collect”. Let’s say we had a task of reading a
structure describing a user from a socket, and have named the operation
“user_read”. Then, the corresponding functions would be named:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>user_read_begin
user_read_collect
</code></pre></div></div>
<p>Which appears to me more obvious than:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>user_read_send
user_read_recv
</code></pre></div></div>
<p>So how does all this help asynchronous programming? The requests help keep the
idea of a process between all those callbacks for various events, since
otherwise those are just functions called from the main loop.</p>
<p>E.g. in an actual request’s “send” (or “begin”) function you would create a
new file descriptor event waiting for input on a socket, and give it a
callback, along with a reference to the request you just created. That
callback would get some data, and then re-add the event, until everything
necessary was read, when it would call “tevent_req_done”.</p>
<p>Furthermore, requests can be nested, i.e. a larger job can be split into
several pieces. No special API is used there, but rather requests just start
new requests in the course of their work, or wait for completion of others.</p>
<p>Overall, tevent requests is a good approach at organizing complicated
processing within an event loop system. It would be even better if the API
used terms closer to the actual purpose and function, and not some (archaic?)
CS terminology requiring imagining what’s not there before you can understand
it.</p>
<p>The event loop approach by itself is very verbose, unfortunately, and requests
play to the same tune: separate function to start work, to collect results,
event callbacks in between, callback for completion, dedicated types to carry
work data between them - all add up to a lot of code which has little to do
with the actual problem being solved.</p>
<p>All that makes me wish for the directness and simplicity of coroutines. C
language doesn’t support them, but there are multiple library implementations
on top, some of which are quite good and even portable. E.g. <a href="http://dotat.at/cgi/~fanf/dotat/~fanf/dotat/git/picoro.git">picoro</a>
from Tony Finch, and a well-explained (but non-portable) attempt by Yossi
Kreinin: <a href="https://www.embeddedrelated.com/showarticle/455.php">Coroutines in one page of C</a>.</p>
<p>If I have to start something complicated and asynchronous in C without using
threads or separate processes, I’ll try coroutines, but for now I have to
understand SSSD and tevent.</p>
<p><strong>UPDATE 06.07.2016</strong>: I had another hypothesis on the origin of the “request”
name and “sending” and “receiving” it, while writing this post. I thought
perhaps these came from the original Samba usecase for it, but decided it was
too quirky and unlikely. Yet, surprisingly (or not, if you wish), it was
indeed the case, as Simo Sorce, a Samba developer among other things,
explained:</p>
<blockquote>
<p>Samba is a file server and the SMB protocol is request driven, and we do a
lot of sending and receiving data over sockets. That’s why as a client you
make a (tevent) request by _send_ing data and then _recv_ing an answer.
Once we established a pattern we decided to use it everywhere, because using
a pattern makes it easy to understand what the code does.</p>
</blockquote>
<p>So, no, this is not archaic CS terminology.</p>
<p>While I understand their original reasoning and intent, I consider the
decision flawed in the long term. My opinion is, in engineering things should
be named according to their essence and function, not a single specific
purpose, so when they’re reused confusion doesn’t ensue. Although, yeah,
hindsight is 20/20, and often it’s hard to perceive the ultimate abstraction
of a particular mechanism and its future uses.</p>
<p>If you’re still struggling with understanding tevent requests, take a look at
the example <a href="https://git.samba.org/?p=samba.git;a=blob;f=lib/tevent/echo_server.c">echo server</a> using them. Along with (neat and
tidy) <a href="https://git.samba.org/?p=samba.git;a=tree;f=lib/tevent">tevent source</a>, it was what eventually and ultimately
cleared my confusion.</p>
Mon, 04 Jul 2016 00:00:00 +0000http://spbnick.github.io/2016/07/04/Making-sense-of-libtevent.html
http://spbnick.github.io/2016/07/04/Making-sense-of-libtevent.htmlBraun Cruzer3 battery replacement<p>About seven years ago I bought a <a href="http://www.service.braun.com/line/SH/S5733/S5733_14_NA.pdf">Braun Cruzer3 shaver</a> and now,
finally, its rechargeable battery started giving up. However, I still had
fresh grille and razors and it was a shame to replace an otherwise
perfectly-functioning device. So I decided to try to replace the batteries.
Here are the steps I took to do it. There are probably not many people in the
world still using these, but perhaps it will help with newer models as well.</p>
<p><strong>DISCLAIMER</strong>: I’m in no way responsible for any damage caused to the shaver,
yourself, your cat, or your house, as a result of following my instructions.
This will also void the device’s warranty. Proceed at your own risk!</p>
<p>To complete this you’ll need:</p>
<ul>
<li>A Torx T8 screwdriver. A flat-blade screwdriver of a certain size could
possibly do as well.</li>
<li>A small piece of flat copper or brass, about half-millimeter thick.</li>
<li>A soldering iron and soldering wire.</li>
<li>Replacement batteries. They must be AAA size, NiMh type. I used Varta,
bought in a general store. The capacity shouldn’t matter, as usual the
bigger the better.</li>
</ul>
<p>Here’s the nervous patient, right before the operation.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/01.before.jpg" alt="Before" /></p>
<p>Remove the screw at the bottom, beside the charger connector.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/02.bottom_screw_in.jpg" alt="Bottom screw in" /></p>
<p>Here it is.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/03.bottom_screw_out.jpg" alt="Bottom screw out" /></p>
<p>Pull the bottom, rubbery cover off. Firmly, but gently. It takes an effort. A
hold as shown on the picture seems to work well. As you do this, a couple of
security tabs break-off somewhere inside, and afterwards the cover slides on
and off easily. Apparently this is how the service would be able to tell
whether you opened the shaver or not. We won’t care about that by now.
As to the cover, the screw will hold it OK.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/04.pulling_the_cover.jpg" alt="Pulling the cover" /></p>
<p>It slides off like this.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/05.cover_slid.jpg" alt="Cover slid" /></p>
<p>And here it’s completely off. You’ll see plenty of hair inside at this moment,
if you used the shaver for a considerable time. I cleaned them off before
taking the picture, to spare you the view.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/06.cover_off.jpg" alt="Cover off" /></p>
<p>Next remove the screws on the sides of the control board housing.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/07.left_housing_screw.jpg" alt="Left housing screw" />
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/08.right_housing_screw.jpg" alt="Right housing screw" /></p>
<p>Here the housing is off. You can see the contacts for the on/off button, the
LED, and the motor, and the rubber seal around them. Pretty neat!
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/09.housing_off.jpg" alt="Housing off" /></p>
<p>You can use a flat screwdriver to help slide the control board out of its
housing, as shown.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/10.pulling_control_board.jpg" alt="Pulling control board" /></p>
<p>Here is the control board, out of its housing.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/11.control_board_out.jpg" alt="Control board out" /></p>
<p>The factory batteries are sitting pretty tight in there (you’ll see why next).
Use a flat screwdriver, or something similar, to get them out.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/12.pulling_first_battery.jpg" alt="Pulling first battery" />
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/13.pulling_second_battery.jpg" alt="Pulling second battery" /></p>
<p>The battery terminals have little round pins welded onto them, which
apparently help hold the batteries firmly against the board’s terminals in the
presence of motor’s vibration.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/14.battery_negative_terminal.jpg" alt="Battery negative terminal" />
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/15.battery_positive_terminal.jpg" alt="Battery positive terminal" /></p>
<p>Here are four of the board’s battery terminals. You can see little round
indentations where the pins are supposed to rest.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/16.control_board_terminals_front.jpg" alt="Control board terminals front" />
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/17.control_board_terminals_back.jpg" alt="Control board terminals back" /></p>
<p>Since we are not likely to get anything like these batteries with pins easily,
we’ll have to do without them. The standard AAA batteries fall out of the
holder easily, and are not likely to have good contact as is. So, I had the
idea to make them fit tighter by soldering a piece of copper to each of the
control board’s negative terminals. Don’t solder anything to the batteries
themselves - you can easily ruin them with heat from the soldering iron!</p>
<p>Here’s a piece of a salvaged copper stock I used.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/18.copper_stock.jpg" alt="Copper stock" /></p>
<p>I cut little squares and bent them in the middle slightly in an attempt to get
a more reliable contact.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/19.copper_contacts.jpg" alt="Copper contacts" /></p>
<p>Here they are soldered in.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/20.soldered_contact_back.jpg" alt="Soldered contact back" />
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/21.soldered_contact_front.jpg" alt="Soldered contact front" /></p>
<p>If you look inside the holder, you should be able to see the polarity marks at
the bottom. Here the plus is visible, and you can just about see the minus.
I chose to solder the copper to the negative terminals, presuming that there
will be more area to make contact with and it will be more reliable.</p>
<p>Observe polarity when inserting the batteries!
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/22.soldered_contacts_both_plus_polarity_marks.jpg" alt="Soldered contacts both plus polarity marks" /></p>
<p>Aaand the new batteries are in.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/23.new_batteries_in.jpg" alt="New batteries in" /></p>
<p>Slide the control board back in. Make sure it goes in straight and easy, don’t
force it. Don’t make the mistake shown on the photo!
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/24.skewed_control_board.jpg" alt="Skewed control board" /></p>
<p>I put a piece of electrical tape on top of the batteries (visible through the
housing) to give them a tighter fit and prevent them from rattling from
motor vibration.</p>
<p>Now the board and the housing are ready to be put back.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/25.control_board_inside.jpg" alt="Control board inside" /></p>
<p>The housing is back on.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/26.housing_in.jpg" alt="Housing in" /></p>
<p>And the rubbery cover as well.
<img src="/assets/2016-06-24-Braun-Cruzer3-battery-replacement/27.cover_on.jpg" alt="Cover on" /></p>
<p>You probably bought your batteries pre-charged, and by now tried turning the
shaver on and it didn’t work. Don’t be alarmed, this is normal. Just plug the
powered charger in for a moment, disconnect, and try again. It should work!
This is likely to not allow the shaver to be turned on with low charge (no
batteries = low charge), so the batteries wouldn’t discharge too much and
leak.</p>
<p>After the replacement, I’ve used the shaver for two (now rather long) recharge
cycles, and saw no problems.</p>
<p>Good luck with your repair!</p>
Fri, 24 Jun 2016 00:00:00 +0000http://spbnick.github.io/2016/06/24/Braun-Cruzer3-battery-replacement.html
http://spbnick.github.io/2016/06/24/Braun-Cruzer3-battery-replacement.htmlOpen-Source Session Recording<p>Many companies need to have their systems used or even managed by people they
don’t entirely trust: contractors, outsourced support, peripheral IT staff,
etc. It helps to know what these users or operators were doing on your
systems, or even what they’re doing right now, so you can not only prevent
repeated issues, but also stop an incident about to happen.</p>
<p>Government, medical and certain other organizations can be required by law to
collect recordings of user sessions. Support desks also appreciate a way to
look back at what exactly happened, so they don’t need to talk through a
user’s recollection of events.</p>
<p>Session recordings are typically required to include whatever happens on the
user’s screen (be it a text terminal or a GUI), what commands or programs the
user executes, what files he/she reads or modifies. They can also include
hosts the user contacts, what URLs he/she opens in the browser, and other
information.</p>
<p>The interface for auditors needs to allow playing back the sessions the way
users saw them, plus searching for and rewinding to particular session events.
It needs to look like the familiar video playback interface with extra
controls. The searchable events are text entered or appearing on the screen,
entered commands, started applications, accessed files, hosts, URLs, etc.</p>
<p>Actual recording implementations range from processes on the session’s host,
through <a href="https://en.wikipedia.org/wiki/Jump_server">jump servers</a>, to <a href="https://en.wikipedia.org/wiki/Application-level_gateway">application-level gateways</a>. The
collected data is stored in a central location, where it can be protected and
easily accessed. The auditor’s interface can be implemented as a standalone
application, or a web UI.</p>
<p>There are many commercial, closed-source solutions from such companies as
<a href="https://www.balabit.com/network-security/scb">BalaBit</a>,
<a href="https://www.centrify.com/products/server-suite/auditing-compliance/session-auditing/">Centrify</a>,
<a href="http://docs.citrix.com/en-us/xenapp-and-xendesktop/7-6/xad-monitor-article/xad-session-recording/xad-sr-get-started.html">Citrix</a>,
<a href="http://www.cyberark.com/solutions/by-project/session-monitoring-and-recording/">CyberARK</a>,
<a href="http://software.dell.com/products/privileged-session-manager/">Dell</a>,
<a href="http://www.observeit.com/product/platforms/unix-linux-monitoring">ObserveIT</a>,
<a href="http://thycotic.com/products/secret-server/features/session-recording/">Thycotic</a>,
<a href="http://www.wheelsystems.com/produkty/fudo/">WHEEL Systems</a>, and others.
Yet, so far there is no integrated open-source solution available, and
that’s exactly what we’re working on right now, at <a href="http://www.redhat.com/">Red Hat</a> <a href="https://access.redhat.com/products/identity-management-and-infrastructure">Identity
Management</a> group.</p>
<p>We decided to start small, reuse as many components as possible, and make it
easy to take apart and reuse in turn. We’re also putting it out in the open,
from the start.</p>
<h2 id="the-stage">The Stage</h2>
<p>Linux already has a very good <a href="https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/chap-system_auditing.html">audit subsystem</a> in the form of <code class="highlighter-rouge">auditd</code>
and the accompanying tools. It can capture most of the required session data.
The audit subsystem can record syscalls done by processes, including files
being accessed or modified and other processes executed. It can also log
network activity when coupled with <code class="highlighter-rouge">iptables</code>. Auditing can be
<a href="http://man7.org/linux/man-pages/man8/auditctl.8.html">configured</a> using a flexible rule system, allowing fine control of
what exactly is recorded and for which users.</p>
<p>Audit subsystem supports <a href="http://man7.org/linux/man-pages/man8/pam_tty_audit.8.html">recording terminal input</a>. However,
it doesn’t support recording terminal output, i.e. what the user actually
sees. Graphical display recording is also far outside its domain.</p>
<p>Any combination of the available logging servers can deliver the session data
(apart from video recordings) to a central location: <a href="http://www.freedesktop.org/software/systemd/man/systemd-journald.service.html">journald</a>,
<a href="http://www.rsyslog.com/">RSYSLOG</a>, <a href="http://www.fluentd.org/">Fluentd</a>, <a href="https://www.elastic.co/products/logstash">Logstash</a>, etc.</p>
<p>There are also a few open-source log aggregation and analysis tools on which
an auditor interface can be built: <a href="https://www.nagios.org/">Nagios</a>,
<a href="https://www.elastic.co/products/elasticsearch">ElasticSearch</a>, <a href="https://www.graylog.org/">Graylog</a>, <a href="https://www.elastic.co/products/kibana">Kibana</a>.</p>
<h2 id="the-plan">The Plan</h2>
<p>We start with implementing recording terminal I/O and other session data. The
graphical display recording presents more challenges and we’ll leave it for
the future.</p>
<p>Adding terminal output recording to audit subsystem both in the kernel and the
user space appears the right thing to do. However, the TTY subsystem, into
which the audit subsystem plugs, is complicated, <code class="highlighter-rouge">auditd</code> might not be able to
handle higher data bandwidth in a timely manner, and arriving to a solution
satisfying everyone can take a long time. So, for the sake of saving time and
having a solution as early as possible, we chose an easier way, and left the
possible proper integration for the future.</p>
<p>We’re implementing a program substituting the user’s login shell, starting the
actual user’s shell under a PTY and logging everything that passes between the
PTY and the actual terminal. The terminal I/O is joining the audit records in
the system log.</p>
<p>All of the above will be set up by <a href="https://fedorahosted.org/sssd/">SSSD</a>, which in turn can be
controlled by <a href="https://www.freeipa.org/">FreeIPA</a>. That includes setting up audit rules and
substituting the shell with the recording program for specific users and
groups. Still, all the “handles” will be exposed for easy integration into
arbitrary systems.</p>
<p>From there, we forward all the collected data to ElasticSearch via any number
of log servers. We will playback the sessions using a purpose-built
visualisation in Kibana.</p>
<p>We chose ElasticSearch for our storage as the most universal, flexible and
scalable solution, with a good following in the community. The choice of
Kibana as the analyzing and visualisation front-end was then straightforward.</p>
<h2 id="the-progress">The Progress</h2>
<p>We have a proof-of-concept terminal I/O recording package implemented, called
<a href="https://github.com/spbnick/tlog"><code class="highlighter-rouge">tlog</code></a>. It can already be used to log I/O to ElasticSearch in a
searchable way, and play it back on a terminal. We’re working on adding
features and polishing it at the moment. See the demo below, or try it
yourself.</p>
<iframe width="740" height="416" src="https://www.youtube.com/embed/F5sOmosbYik?rel=0&amp;showinfo=0" frameborder="0" allowfullscreen="">
</iframe>
<p>The demo shows recording of a user’s session to ElasticSearch with
simultaneous playback, recorded messages as seen in Kibana, and also
highlights the effect of ElasticSearch streaming limitations (and simplicity
of <code class="highlighter-rouge">tlog</code>’s current implementation, of course).</p>
<p><code class="highlighter-rouge">Tlog</code> is still in the early stages of development, there are plenty of items on
the <a href="https://github.com/spbnick/tlog/blob/master/todo.txt">TODO list</a> and we’ll be adding more, without doubt. We’ll be
glad to receive contributions, suggestions, questions or just comments. Please
<a href="https://github.com/spbnick/tlog">fork</a>, submit pull requests, issues, or write <a href="mailto:spbnick@gmail.com?subject=tlog">me</a> directly!</p>
<h2 id="the-limitations">The Limitations</h2>
<p>Since humans type slowly and the I/O recording needs to be chopped into log
messages, a smoothing delay needs to be introduced to avoid creating a
separate message for each or just a few keystrokes. This introduces latency.
Delivery via a number of log servers, plus ElasticSearch indexing make it
longer. This makes implementing real-time terminal monitoring problematic.</p>
<p>However, most of the time, a real person will not be watching a live stream
from a user’s terminal, but instead an automatic system will be watching the
stream for patterns, or a recording will be reviewed later. Plus, other
session data doesn’t need to have a smoothing delay and e.g. commands and
accessed files will be delivered faster.</p>
<p>Even though the text on a terminal looks uninterrupted, the actual recorded
I/O stream is often interspersed with control codes: color, font changes and
cursor movements in the output, editing commands and other special characters</p>
<ul>
<li>in the input. E.g. while a user enters a specific valid command on a shell
command line, the input stream in the recording will be very different, if the
command was edited before being executed - a common occurrence.</li>
</ul>
<p>Moreover, since the I/O stream needs to be chopped into messages, a particular
word or phrase can end up split between two messages and thus impossible to
find as a whole using ElasticSearch facilities. For these reasons, the
structured audit records for specific events should be searched instead, and
the I/O stream should be searched only for the data unavailable otherwise, and
the results interpreted carefully.</p>
<p>ElasticSearch is <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/modeling-your-data.html">not a relational database</a> and it
doesn’t support joining different data types in search queries as would be
possible with SQL. This means that data structures need to be denormalized,
i.e. a container needs to be stored in a single document with its items, and
data belonging to many containers needs to be copied to each of them. For I/O
recording it means that there can’t be a single document describing the
session as a whole, and separate documents describing pieces of the stream.
Instead, overall session data needs to be attached to each piece. This
introduces size overhead to each of the pieces.</p>
<p>However, if the maximum I/O message size is set high enough, this overhead
becomes negligible. The alternative is to join documents client-side, which is
inefficient for smaller documents, but might still be necessary in some cases.
This also concerns multi-part audit events.</p>
<h2 id="the-challenges">The Challenges</h2>
<p>Among the more interesting challenges implementing session recording as
planned are:</p>
<ul>
<li>
<p>Nobody tried to implement converting audit messages to
ElasticSearch-compatible JSON, yet. It should be possible and not very hard,
as audit messages are relatively well-structured.</p>
</li>
<li>
<p>So far, ElasticSearch has very limited support for streaming new documents.
Any stream-oriented client, such as a terminal I/O player, needs to resort
to inefficient polling, or possibly difficult to use unofficial plug-ins.</p>
</li>
<li>
<p>Nobody tried making anything like the terminal I/O playback visualisation we
plan to implement for Kibana. It is not yet known if it’s doable, and while
theoretically it is possible, it is unlikely to be easy.</p>
</li>
</ul>
<h2 id="the-future">The Future</h2>
<p>Our target is a turn-key solution: install the FreeIPA client, turn a few
knobs on FreeIPA server, and have sessions recorded for specific users or
groups. Log delivery and storage will have to be configured separately,
though, and we’re working on that in another project.</p>
<p>Beside basic playback we plan to have the Kibana visualisation provide
accelerated playback, searching for and rewinding to entered/printed text,
executed processes, accessed files and more. We’ll have this contributed
upstream, or available as a plug-in. We’re considering embedding the
visualisation into FreeIPA web UI to enable viewing specific sessions with
controls to terminate/suspend them.</p>
<p><a href="/feed.xml">Stay tuned</a> for more posts on session recording, <code class="highlighter-rouge">tlog</code> and other tech
topics!</p>
Mon, 26 Oct 2015 15:30:00 +0000http://spbnick.github.io/2015/10/26/open-source-session-recording.html
http://spbnick.github.io/2015/10/26/open-source-session-recording.htmlPace DIT7100/14 STB teardown<p>Elisa, a Finnish telecom company is phasing out one of its older STBs (set-top
boxes) and they are already showing up in local reuse centers
(<a href="http://www.kierratyskeskus.fi/">Kierrätyskeskus</a>, an analogue of US and Canada’s Goodwill
Stores) and flea markets. The particular model I happened to have is Pace
DIT7100/14. These seem to be relatively powerful machines, on par with the
popular <a href="https://www.raspberrypi.org/products/model-b/">Raspberry Pi Model B</a>, but cheaper, selling at about 15
EUR, and I suspect can be acquired for less. So, they can theoretically be put
to a few good uses beside their primary purpose.</p>
<p><a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/top.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/top.thumb.jpg" alt="Top" title="Top" /></a> <a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/bottom.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/bottom.thumb.jpg" alt="Bottom" title="Bottom" /></a> <a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/rear.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/rear.thumb.jpg" alt="Rear" title="Rear" /></a></p>
<!--more-->
<p>I haven’t found any evidence of anyone (beside Pace) successfully putting
Linux on those, and so I cracked mine open to see if it is at least possible.
Unfortunately, I wasn’t able to discern much, or even find the serial console
port yet.</p>
<p>The STB has a number of ports. Front: 1 USB socket, 1 infrared receiver. Back:
the tuner’s RF in and out, SPDIF optical audio, analog audio L/R, HDMI, SCART,
Ethernet’s RJ-45, another USB socket and 12V DC input.</p>
<p>Removing the top cover we can see the main board and the controls board
mounted above it.</p>
<p><a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/top_open.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/top_open.thumb.jpg" alt="Top (open)" title="Top (open)" /></a></p>
<p>The controls board contains the soft power button, status LED, IR receiver,
front USB socket. Also, amusingly, <a href="http://www.ti.com/product/cc2533">TI’s CC2533</a> - a 2.4GHz IEEE
802.15.4/ZigBee SoC, along with its 32 MHz oscillator, even though the STB
shipped with only an <a href="http://www.uei.com/product/europe-middle-east-africa/widor">IR remote</a>.</p>
<p><a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/controls_board_top.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/controls_board_top.thumb.jpg" alt="Controls board (top)" title="Controls board (top)" /></a>
<a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/controls_board_bottom.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/controls_board_bottom.thumb.jpg" alt="Controls board (bottom)" title="Controls board (bottom)" /></a></p>
<p>Moving onto the most interesting part - the main board, we can find the CPU
under a heatsink. Removing the heatsink reveals the CPU being Broadcom’s
BCM7406. I couldn’t find much information on it, but I suspect it’s closely
related to <a href="http://www.broadcom.com/products/set-top-box-and-media-processors/cable-%28stb-and-media-processors%29/bcm7405">BCM7405</a> for which
<a href="http://hwdb.mipt.cc/BCM7405">some information</a> can be found on the wonderful
<a href="http://hwdb.mipt.cc/">Programmer’s Hardware Database wiki</a>.
Including the <a href="http://hwdb.mipt.cc/BCM7405/Pinout">pinout</a>. LinuxGizmos.com also has an
<a href="http://archive.linuxgizmos.com/65nm-stb-on-a-chip-runs-linux/">article</a> in its archive introducing 7405 with the
function block and reference design architecture diagrams.</p>
<p><a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/main_board_top.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/main_board_top.thumb.jpg" alt="Main board (top)" title="Main board (top)" /></a>
<a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/main_board_bottom.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/main_board_bottom.thumb.jpg" alt="Main board (bottom)" title="Main board (bottom)" /></a>
<a href="/assets/2015-06-19-pace-dit7100-14-STB-teardown/cpu.jpg"><img src="/assets/2015-06-19-pace-dit7100-14-STB-teardown/cpu.thumb.jpg" alt="CPU (without heatsink)" title="CPU (without heatsink)" /></a></p>
<p>Going further, the main board contains four <a href="http://www.samsung.com/global/business/semiconductor/file/product/ds_k4t1gxx4qf_rev12-0.pdf">1Gb DDR2 SDRAM chips from
Samsung</a>, two on each side. This amounts to 512 MB of RAM.
There’s also what seems to be a <a href="http://www.spansion.com/Support/Datasheets/S29GL-P_00.pdf">Spansion 1Gb NOR flash chip</a>
labeled GL01GP11FFSS8. I.e. there’s only 128 MB of storage on board, which is
not too much, but USB ports can help. Then there’s <a href="http://www.sony.net/Products/SC-HP/cx_news_archives/img/pdf/vol_60/cxd2820r.pdf">CXD2820R</a> -
a DVB-T2 demodulator from Sony, and <a href="http://www.st.com/web/catalog/mmc/FM131/SC631/PF174515">STV6417</a> - an A/V switch from
ST. The rest of the chips seems to be logic glue or power-related.</p>
<p>As I said above, I wasn’t able to find the serial console. I poked around the
connector pads to the left of the CPU with a scope during boot-up and didn’t
find anything. The major suspect - a four-hole space for a connector had two
grounds in the middle and nothing on the sides, it seems. The 23- and 15-pad
spaces were also mostly silent with some clock on one or two pads.</p>
<p>From reading about getting serial console on other similar devices I gathered
they can be actually routed to some outside connectors and/or require some
buttons pressed to be activated during boot. I tried the only button - Power
and couldn’t find anything on the unpopulated pads anyway, but I haven’t tried
the connectors. I’ll try looking at the pinout and muxing and see if UART can
be shared with some of the pins possibly used by the connectors directly.</p>
<p>I haven’t tried locating JTAG yet, mostly because I don’t have an EJTAG cable.
It can also be disabled. Lastly, it is quite possible that the device was
Tivoized in some way, making any attempts to hack it useless, but I have to
hope for the best until I’m able to figure it out.</p>
<p>If you have any ideas of where the serial or JTAG can be on this board, or if
this STB is Tivoized, please write to me, or leave a comment on my <a href="https://plus.google.com/+NikolaiKondrashov/posts/SVHbiedQPhy">Google+
post</a>.</p>
Fri, 19 Jun 2015 00:00:00 +0000http://spbnick.github.io/2015/06/19/pace-dit7100-14-STB-teardown.html
http://spbnick.github.io/2015/06/19/pace-dit7100-14-STB-teardown.htmlGetting interactive shell in the deepest bowels of a system<p>Sometimes you might need to debug a failure somewhere in the deepest bowels of
a system. Particularly a complicated shell script or a build system, where
execution is very much affected by the state of and location in the filesystem
and environment variables. In these cases getting an interactive shell in the
exact problematic environment might be very useful.</p>
<p>With Bash scripts that run interactively it might be sufficient to just put
this command at the desired “breakpoint”:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash -rcfile &lt;(set)
</code></pre></div></div>
<p>This starts an interactive shell with the invoking shell’s functions and
variables. If you need variable attributes (including “export” status) and
various shell options transferred as well, then this might do the job:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>bash -rcfile &lt;(declare -p; declare -f; set +o; shopt -p)
</code></pre></div></div>
<p>However, sometimes the script or a build system might have no access to a
terminal. Then <code class="highlighter-rouge">screen</code> comes to the rescue. This command:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>screen -D -m
</code></pre></div></div>
<p>will start a detached screen session and will wait until it is finished, so
that you can connect to it with <code class="highlighter-rouge">screen -r</code> and get your interactive shell.</p>
<p>You can combine this and either of the previous approaches, but shell state
will need to be passed via a temporary file or a named pipe, as process
substitution’s file descriptors won’t be inherited by the shell inside
<code class="highlighter-rouge">screen</code>:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>set &gt;tmp_rcfile; screen -D -m bash -rcfile tmp_rcfile
</code></pre></div></div>
<p>Note though, that <code class="highlighter-rouge">screen</code> is a setuid executable, so it removes some
variables from the environment, such as LD_PRELOAD and LD_LIBRARY_PATH, for
safety. You will need to pass them explicitly. For example, like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>screen -D -m bash -c "LD_PRELOAD='$LD_PRELOAD' bash"
</code></pre></div></div>
Tue, 13 Jan 2015 18:55:59 +0000http://spbnick.github.io/2015/01/13/getting-interactive-shell-in-the-deepest-bowels-of-a-system.html
http://spbnick.github.io/2015/01/13/getting-interactive-shell-in-the-deepest-bowels-of-a-system.htmlLiquid: truncate without dots<p>While building <a href="http://digimend.github.io">the new DIGImend website</a> I needed a way to truncate
strings in <a href="http://docs.shopify.com/themes/liquid-documentation/basics">Liquid</a>, which is the template language
<a href="http://jekyllrb.com">Jekyll</a> uses. Specifically, I needed to compare parts of page URLs
when iterating over sub-pages of a page to generate menus and page lists.</p>
<p>To my surprise I couldn’t find any. The version used by Jekyll didn’t seem to
support <a href="http://docs.shopify.com/themes/liquid-documentation/filters/string-filters#slice"><code class="highlighter-rouge">slice</code></a>, and the only alternative,
<a href="http://docs.shopify.com/themes/liquid-documentation/filters/string-filters#truncate"><code class="highlighter-rouge">truncate</code></a>, seemed to always replace the end of the
truncated string with three dots (“…”, an ellipsis).</p>
<p>However, diving into Liquid source in search of an alternative, I found that
<a href="http://docs.shopify.com/themes/liquid-documentation/filters/string-filters#truncate"><code class="highlighter-rouge">truncate</code></a> accepts optional second argument: a string to
replace that ellipsis with. So I could write this:</p>
<figure class="highlight"><pre><code class="language-liquid" data-lang="liquid"><span class="p">{%</span><span class="w"> </span><span class="nt">assign</span><span class="w"> </span><span class="nv">page_base</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">page_dir</span><span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="nf">truncate</span><span class="p">:</span><span class="w"> </span>parent_dir_len<span class="p">,</span><span class="w"> </span><span class="s2">""</span><span class="w"> </span><span class="p">%}</span></code></pre></figure>
<p>When writing this post I found that the Liquid’s GitHub project wiki
<a href="https://github.com/Shopify/liquid/wiki/Liquid-for-Designers#standard-filters">mentions</a> this argument, but being otherwise quite low
on details it wasn’t the first place to look to me.</p>
Tue, 07 Oct 2014 19:18:00 +0000http://spbnick.github.io/2014/10/07/liquid-truncate-without-dots.html
http://spbnick.github.io/2014/10/07/liquid-truncate-without-dots.html