dpavlin's Journalhttp://use.perl.org/images/topics/useperl.gifhttp://use.perl.org/~dpavlin/journal/
irc-logger - memory augmentation for #irchttp://use.perl.org/~dpavlin/journal/35887?from=rss
<p>Initially <a href="http://svn.rot13.org/index.cgi/irc-logger/revision/?rev=4">created in 2006</a> this handy tool is best described with original commit message:</p><blockquote><div><p>IRC bot which replace human memory</p></div></blockquote><p>Here is a quick run-down through available features:</p><ul>
<li>web archive with search</li><li>irc commands: last, grep/search, stat, poll/count</li><li> <tt>tags//</tt> in normal irc messages (tagcloud, filter by tag, export as RSS feed)</li><li>announce<nobr> <wbr></nobr><tt>/me</tt> messages to <a href="http://twitter.com/dpavlin">Twitter</a> (yes, lame, but that was <a href="http://svn.rot13.org/index.cgi/irc-logger/revision/?rev=53">a year ago</a>)
</li><li>tags are available as html links for embedding (in wikis)</li><li>RSS feed from messages with tags (also nice for embedding)</li><li>irssi log import (useful for recovery in case of failure of machine or service<nobr> <wbr></nobr>:-)</li><li>announce new messages from RSS feeds (nice for wiki changes, blog entries or commits)
</li></ul><p>It has grown quite a bit from initial vision to recall last messages on the web (and it does go through some hoops to produce nice web archive). Adding of tags allowed easy recall of interesting topics but in a way now it provides an central hub for different content connected to irc.</p><p>It's written in perl using POE and it's probably not best example of POE usage. It is also somewhat PostgreSQL specific but works well for our small community at <tt>#razmjenavjestina</tt> irc channel. Since I have seen some interest in it this blog post might serve as announce of it's existence.</p><p>I will probably add some documentation to <a href="http://wiki.rot13.org/rot13/index.cgi?irc-logger">it's wiki page</a> and add real muti-channel support (most of code is in there, but web archive needs filtering by channel). If you are interested to<nobr> <wbr></nobr><tt>/invite</tt> it to your channel, drop me a note.</p><p>
<small>Originally I wrote this <a href="http://blog.rot13.org/2008/03/irc-logger_-_memory_augmentation_for_irc.html">a couple days ago</a>, but I made some more progress and code is now no longer so ugly that I can't share it with other perlers, so here it is...
</small></p>dpavlin2008-03-11T23:35:59+00:00journalCWMP and MDAP servershttp://use.perl.org/~dpavlin/journal/34918?from=rss
<p>After nine months of playing with Thompson ADSL modems I have two projects which I wrote in perl, both of which is, as far as I know, first Open Source GPL implementation of those protocols.</p><p>
<b>MDAP</b>
</p><p>MDAP is protocol used by Thompson CPE devices to issue commands to CPEs (called ants) using multicast address <tt>224.0.0.103</tt> and port <tt>3235</tt> registered by IANA.</p><p>It's very cool idea, since you can connect as many devices as you have network ports or bandwidth, and allthough they all will boot with same IP address (and this create conflicts on IP network), you can still sand commands to each individual device using multicast.</p><p>Originally I developed it to flash multiple modems at once, but since I also added simple rules to change IP addresses and issue commands to devices. This essentially enables you to flash devices to some version of firmware and then change configuration a bit and have you test lab ready in few moments.</p><p>This project doesn't have a real project page yet, but you can <a href="http://svn.rot13.org/index.cgi/mdap">take a look at source</a> if you are interested...</p><p>This project also include (but doesn't use yet) simple perl BOOTP and TFTP servers, so in the end it will probably have perl-only solution for MDAP. If you just use included scripts and documentation for setup, it will use binary bootp and tftp server since this configuration was in use for at least half of the year, and I consider it stable.</p><p>
<b>CWMP</b>
</p><p> <a href="http://code.google.com/p/perl-cwmp/">Perl CWMP server</a> is essentially a low-level support for broken idea to communicate with devices using persistent connection SOAP with invalid XML (name spaces in some responses are just invalid) known as TR-069.</p><p>This is work-in-progress, and right now it's stable enough to work with multiple devices at once. In essence, it's protocol-violating SOAP server implementing persistent connection handling as described in TR-069 documentation (empty post, even without headers as first request, ehhh....).</p><p>Idea is to enable you, the user, to write perl rules against CPE devices.</p><p>It's half-way there: It does have disk-based command queue for each device (which is also NFS save, which is nice if you want to have multiple servers) and persistent storage for each CPE internal data tree implemented using DBM::Deep or YAML. When used with YAML, it great way to understand protocol. However, not all methods are implemented, but I hope to have full implementation by end of January 2008.</p>dpavlin2007-11-18T13:37:05+00:00journalExhibit facet browsinghttp://use.perl.org/~dpavlin/journal/34204?from=rss
We have few mp3 players which no longer work, but are still under warranty. So idea was to pick another device (which will hopefully work longer). However, on-line shops leave a lot to be desired if you want to just do quick filtering of data.
<p>
As a very fortunate incident, I stumbled upon <a href="http://simile.mit.edu/exhibit/">Exhibit</a> from <a href="http://simile.mit.edu/">SMILE</a> project at MIT which brought us such nice tools as <a href="http://simile.mit.edu/timeline/">Timeline</a> and <a href="http://simile.mit.edu/potluck/">Potluck</a>.
</p><p>
So, I scraped web, converted it to CSV and tried to do something with it. In the process I again re-visited the problem of semi-structured data: while data is separated in columns, one column has generic description, player name and all characteristics in it.
</p><p>
So, what did I do? Well, I started with CPAN and few hours later I had a <a href="http://svn.rot13.org/index.cgi/simile/view/links/csv2js.pl">script which is rather good in parsing semi-structured CSV files</a>. It supports following:
</p><ul>
<li>guess CSV delimiter on it's own (using <a href="http://search.cpan.org/~enell/Text-CSV-Separator/"> <tt>Text::CSV::Separator</tt> </a>)</li><li>recognize 10 Kb and similar sizes and normalize them (using <a href="http://search.cpan.org/~ferreira/Number-Bytes-Human/"> <tt>Number::Bytes::Human</tt> </a>)</li><li>splitting of comma (<tt>,</tt>) separated values within single field</li><li>strip common prefix from all values in one column</li><li>group values and produce additional properties in data</li><li>generate specified number of groups for numeric data, useful for price ranges</li><li>produce JSON output for Exhibit using <a href="http://search.cpan.org/~audreyt/YAML-Syck/"> <tt>JSON::Syck</tt> </a> </li></ul><p>
<a href="http://blog.rot13.org/demo/links/links.html">
So how does it look?</a>
</p><p>
In the end, it is very similar to the way <a href="http://www.dabbledb.com/">Dabble DB</a> parses your input. But, I never actually had any luck importing data into Dabble DB, so this one works better for me<nobr> <wbr></nobr><tt>:-)</tt>
</p><p>
This will probably evolve to universal munger from CSV to arbitrary hash structure. What would be good name? <tt>Text::CSV::Mungler</tt>?
</p><p>
This is a first post in <a href="http://blog.rot13.org/hackoftheweek/">series of posts</a> which will cover one hack a week on my blog. This will (hopefully) force me to write at least one post a week on one side, and provide some historic trace about my work for later.</p>dpavlin2007-08-24T09:12:20+00:00journalSearch::Estraier 0.07 now available on CPANhttp://use.perl.org/~dpavlin/journal/30544?from=rss
<p>After several months of testing, several new releases of Hyper Estraier with new features (newest one is masking [excluding] of linked databases when searching) version 0.07 is finally ready.</p><p>I would really suggest to all current users to upgrade to latest version. It fixes problems with set_skip and vectors and has, well, set_mask.</p><p>Require explicit version of this module when using new features, just to be safe that you are not running against older version (it's only sane behavior).</p>dpavlin2006-08-06T19:28:08+00:00journalSearch::Estraier 0.06 covers whole P2P APIhttp://use.perl.org/~dpavlin/journal/29580?from=rss
<p>I'm somewhat proud to annouce that new version of Search::Estraier now supports master node commands, thus brining this implementation in sync with P2P documentation of current Hyper Estraier version 1.2.5.</p>dpavlin2006-05-10T17:26:27+00:00journalBiblio::Isishttp://use.perl.org/~dpavlin/journal/22838?from=rss
<p>As I figured out by now, people are not reading journals with single post in them.</p><p>I as too lazy to move my perl-related writing here (mostly because I write them off-line) but I should note that I actually wrote perl-only replacement for OpenIsis called Biblio::Isis. It's on CPAN.</p>dpavlin2005-01-22T18:20:46+00:00journalStrict(ly) perl journalhttp://use.perl.org/~dpavlin/journal/17951?from=rss
So, I gathered enough courage to actually write my first post on <tt>use Perl;</tt>. I will try to post only perl related items here (and reference them from my <a href="http://blog.rot13.org/">primary weblog</a>).
<br>
For a start, I'm writing perl-only module to read ISIS. It's a fun project which started out of frustration with <a href="http://www.openisis.org/">OpenIsis</a> perl bindings. It does read Isis files correctly but not IsisMarc, so I'm not releasing it yet. Is there any interest in it, and what would be proper CPAN namespace for it?dpavlin2004-03-18T18:41:06+00:00journal