Michał Wojciechowskitag:blogs.perl.org,2009-11-03:/users/michal_wojciechowski//9542013-10-23T02:51:59ZA blog about the Perl programming languageMovable Type Pro 4.38POD Web Viewtag:blogs.perl.org,2013:/users/michal_wojciechowski//954.52632013-10-23T02:48:53Z2013-10-23T02:51:59ZWhen developing a Perl module, I often want to get a quick preview of the documentation that I'm writing, just to see if everything is in order and how it turns out. I used to do this the old fashioned...Michał Wojciechowskihttp://odyniec.net/
When developing a Perl module, I often want to get a quick preview of the documentation that I'm writing, just to see if everything is in order and how it turns out. I used to do this the old fashioned way, by generating an HTML file with pod2html or pod2cpanhtml and opening it in a browser, but I was hoping in this day and age there is an easier and better solution, preferably a web application.

Looking around, however, the only thing I could find was the pod2html page at the CPAN Search site, which allows you to upload a POD file, have it processed by pod2html, and displayed with CPAN style. I thought it might be a good idea to try building something more user-friendly, with features like editing POD in the browser, drag and drop file uploads, etc.

And what better time for a little project like this than a weekend when you're ill and not supposed to leave your apartment? Well, that's what my last weekend was like -- two days of coughing and coding, and here's the result: POD Web View.

]]>
The application allows you to upload a POD file, get it from a URL, or paste its contents and edit it on the fly. The generated HTML can be displayed in the style of your choice, mimicking how it would look on CPAN, MetaCPAN, or GitHub.

I hope this will be useful for at least a few fellow Perl developers, like it already is for me. Please note that at this point this is still work in progress -- the backend code needs some more work (e.g. basic sanity checks), and there are a couple UI issues that I'm aware of (and likely a dozen more that I'm not). Anyway, be my guest and give it a try, and if you'd like to report an issue, or maybe help me with the development (more than welcome), I've put the project up on GitHub.

]]>
HTTP::Tiny::SPDY - A Subclass of HTTP::Tiny with SPDY Supporttag:blogs.perl.org,2013:/users/michal_wojciechowski//954.52272013-10-14T23:04:59Z2013-10-14T23:18:31ZI have recently returned to working on Arriba, the PSGI-compliant web server with support for the SPDY protocol, modeled on Starman. I sort of released it (on GitHub only, not on CPAN) back in January, at that time the code...Michał Wojciechowskihttp://odyniec.net/
I have recently returned to working on Arriba, the PSGI-compliant web server with support for the SPDY protocol, modeled on Starman.

I sort of released it (on GitHub only, not on CPAN) back in January, at that time the code was passing the tests in Plack::Test::Suite when running as a regular HTTP/HTTPS server. My next goal, before considering the module ready to be released on CPAN, was to make it pass those tests in SPDY mode. This meant I needed to add support for SPDY to good old LWP::UserAgent, which was used as the HTTP client in Plack tests.

Over the weeks/months that followed, I made a few attemps at tackling this problem, but had a hard time wrapping my head around the architecture of LWP::UserAgent and figuring out a reasonable way to add SPDY into the mix. Having very little time to devote to this project, I didn't get anywhere with it.

]]>
A few days ago, I was delighted to find out that the problem went away by itself, since Plack switched from using LWP::UserAgent to its own Plack::LWPish, which is built around HTTP::Tiny. Now I needed to implement SPDY in HTTP::Tiny, which is, well, tiny when compared to LWP::UserAgent, so the task seemed much easier. I gave it a shot this weekend and got it working in a matter of hours, spawning HTTP::Tiny::SPDY, a subclass of HTTP::Tiny that works the same as the original, but can also do SPDY.

I immediately used the module for the intended purpose of testing Arriba in SPDY mode, and, as expected, this revealed many problems, but most of them turned out to be easy to fix (except for one, which took me more than three hours just because I didn't RTFM in the first place -- will I ever learn?). Soon, Arriba running SPDY was passing all the tests in the suite, which I happily celebrated with a tasty porter beer. I am now cleaning up the code to prepare it to be finally released on CPAN.

And speaking of CPAN, HTTP::Tiny::SPDY is already there, as well as on GitHub. Like Arriba, this is an early release, the code is hackish and immature, and I take zero responsibility for the pain and suffering that you may bring upon yourself when you try to use it. But if you do, I crave your feedback.

]]>
Arriba - PSGI Web Server with SPDY Supporttag:blogs.perl.org,2013:/users/michal_wojciechowski//954.41842013-01-08T20:46:26Z2013-01-09T00:27:33ZHaving some free time during the Christmas break, I decided to give a shot at a small project that I had in mind for a while, and that was to develop a PSGI-compliant(-ish) web server with support for the SPDY...Michał Wojciechowskihttp://odyniec.net/
Having some free time during the Christmas break, I decided to give a shot at a small project that I had in mind for a while, and that was to develop a PSGI-compliant(-ish) web server with support for the SPDY protocol.

In case you haven’t heard of it, SPDY is a networking protocol developed at Google with a goal of reducing web page load latency. It is currently used by some of Google services (including search and Gmail) and by Twitter, and is supported natively in Firefox, Chrome, and Opera — so if you visited any of those sites with any of those browsers, it’s highly likely that your web content was transmitted by means of SPDY. An official standard for the protocol is in the works.

]]>
There was a SPDY module on CPAN that looked promising — Net::SPDY by Lubomir Rintel. While not being a complete implementation of the protocol, it seemed to be working, as I found out by playing with the sample client and server scripts included in the distribution.

After a few days of reading the SPDY specs, minor reverse engineering of other implementations, and blatantly copying (a lot of) code from Starman, I was able to put together a preforking web server operational enough to run a few simple Dancer applications. It’s a mess and nowhere near being ready for production use, but I’m happy to share it to maybe get some feedback from you fine folks — I’ve put it up on GitHub. I intend to continue working on it and hopefully one day turn it into something half-decent.

If you want to run it, be aware that you currently need to use the Net::SPDY module from my forked repository instead of the original one, since in the original there’s some test code that breaks normal server communication.

About the project name — I followed the idea of using friendly names like Starman and Twiggy, and since SPDY reminds me of Speedy Gonzales, I used a part of Speedy’s catch phrase (“¡Ándele! ¡Ándele! ¡Arriba! ¡Arriba!”). However, I know I’m terrible at naming things, so I’m open to suggestions for a better name.

]]>
Building a Search Web App with Dancer and Sphinxtag:blogs.perl.org,2012:/users/michal_wojciechowski//954.41292012-12-14T12:43:57Z2012-12-14T14:16:19ZIn this article, we'll develop a basic search application using Dancer and Sphinx. Sphinx is an open source search engine that's fairly easy to use, but powerful enough to be deployed in high-traffic sites, such as Craigslist and Dailymotion. In...Michał Wojciechowskihttp://odyniec.net/
In this article, we'll develop a basic search application using Dancer and
Sphinx. Sphinx is an open source search engine
that's fairly easy to use, but powerful enough to be deployed in high-traffic
sites, such as Craigslist and Dailymotion.

In keeping with this year's Dancer Advent Calendar trend, the example app will
be built on Dancer 2, but it should work just as well with Dancer 1.

Alright, let's get to work.

]]>
The Data

Our web application will be used to search through documents stored in a MySQL
database. We'll use a simple table with the following structure:

Each document has an unique ID, a title, and contents, stored as both plain text
and as HTML. We need the two formats for different purposes -- HTML will be used
to display the document in the browser, while plain text will be fed to the
indexing mechanism of the search engine (because we do not want to index the
HTML tags, obviously).

We can populate the database with any kind of document data -- for my test
version, I used a simple script to fill the database with POD documentation
extracted from Dancer distribution. The script is included at the end of this
article, in case you'd like to use it yourself.

When Sphinx is installed, it needs to be configured before we can play with it.
Its main configuration file is usually located at /etc/sphinx/sphinx.conf.
For our purposes, a very basic setup will do -- we'll put the following in the
sphinx.conf file:

This defines one source, which is what Sphinx uses to gather data, and one
index, which will be created by processing the collected data and will then
be queried when we perform the searches. In our case, the source is the
documents database that we just created. The sql_query directive defines the
SELECT query that Sphinx will use to pull the data, and it includes all the
fields from the documents table, except contents_html -- like we said,
HTML is not supposed to be indexed.

That's all that we need to start using Sphinx. After we make sure the searchd
daemon is running, we can proceed with indexing the data. We call indexer
with the name of the index:

$ indexer test

It should spit out some information about the indexing operation, and when it's
done, we can do our first search:

It's the documentation for Dancer::Plugin, and it makes total sense that this
is the first result for the word plugin. Sphinx setup is thus ready and we
can get to the web application part of our little project.

The Basic Application

We'll start with a simple web application (let's call it DancerSearch) that
just shows a search form, and then we'll extend it with more features. It will
be using Dancer 2.0, and the Dancer::Plugin::Database plugin (we'll use it to
access the documents database). The code below is the initial
lib/DancerSearch.pm file:

Last but not least, we need a configuration file to tell our app which layout we
want to use, and how to connect to our documents database using the
Dancer::Plugin::Database plugin. This goes into config.yml:

We can now launch the application, and it will greet us with a search form.
Which, unsurprisingly, doesn't work yet. Let's wire it up to Sphinx.

The Sphinx::Search CPAN Module

There is a CPAN module called Sphinx::Search that provides a Perl interface
to Sphinx, and we're going to use it in our app. We put use Sphinx::Search in
DancerSearch.pm, and add the following piece of code before the get '/'
route handler:

This creates a new instance of Sphinx::Search (which will be used to talk to the
Sphinx daemon and do the searches), and sets up a few basic options, such as how
many results should be returned and in what order. Now comes the most
interesting part -- actually performing a search in our application. We insert
this chunk of code at the beginning of the get '/' route handler:

Let's go through what is happening here. First, we check if there was actually a
search phrase in the query string (params('query')->{'phrase'}). If there
was one, we pass it to the $sph->Query() method, which queries Sphinx and
returns the search results (the returned data structure is briefly explained in
the description of the Query method in Sphinx::Search documentation).

We then check the number of results ($results->{'total_found'}), and if
it's greater than zero, it means we found something and we need to retrieve the
documents data from the database. Sphinx only returns the IDs of the matching
documents (as shown earlier in the test search that we did using the command
line), so we need to send a query to the database to get the actual data, such
as document titles that we want to display in the results (note that we're using
the ORDER BY FIELD construct in the SELECT query to maintain the same
order as the list returned by Sphinx).

When we have the documents data ready, we pass it along with other information
(such as the total number of results) to be displayed in our index template.
But, hold on a second -- the template is not yet ready to display the results,
it only shows the search form. Let's fix that now -- below the search form, we
add the following code:

This displays the phrase that was submitted, the number of hits, and a list of
results (or a "no hits" message if there weren't any).

And you know what? We're now ready to actually do a search in the browser:

Neat, we have a working search application! We're just missing one important
thing, and that is being able to access a document that was found. The results
link to /document/:document_id, but that route isn't recognized by our app.
No worries, we can fix that easily:

This route handler is pretty straightforward, we grab the ID from the URL, use
it in a SELECT query to the documents table, and return the HTML contents of
the matching document (or a 404 page, if there's no document with that ID).

Conclusion

What we've built is still a very basic application, lacking many features -- the
most obvious one that's missing is pagination, and being able to access results
further down the list, not just the first ten. However, the code can be easily
extended, thanks to the flexibility and ease of use of both Dancer and Sphinx.
With a bit of effort, it can be made into an useful search app for a knowledge
base site, or a wiki.

I think this application is a good example of how Dancer benefits from being
part of the Perl ecosystem, giving web developers the ability to make use of the
thousands of modules in CPAN (like we just did with Sphinx::Search). This allows
to build working prototypes of web applications and implement complex features
in a very short time.

The POD Extraction Script

As promised, this is the script that I used to extract the POD from Dancer
distribution and store it in the MySQL database:

You can run it with one argument, which is the location of the directory that
will be scanned (recursively) for .pm/.pod files, or with no arguments, in which
case the script will work with the current directory.

(Note: The script makes use of Pod::Simple, which I'm not very familiar with,
so it's possible that I'm doing something stupid with it -- if that's the case,
please let me know.)

]]>
Let Paths Be Paths Againtag:blogs.perl.org,2012:/users/michal_wojciechowski//954.28732012-02-26T22:29:31Z2012-02-26T22:41:26ZMichał Wojciechowskihttp://odyniec.net/
The de facto standard way of constructing portable filesystem paths in Perl is through the use of File::Spec's catfile and catdir functions. Example:

my $path = File::Spec->catfile('dir', 'subdir', 'file.txt');

This method, or a similar one involving Path::Class, is the most recommended approach and has been adopted by application development frameworks like Dancer (which has a wrapper method for it, named path) and Catalyst (with its path_to method).

The slight problem that I see with this method is that it makes code a bit more complicated, and thus a bit less readable. Paths become lists of parameters and no longer look like paths.

I wrote a simple module that tries to address this by allowing you to write paths the traditional way -- as strings, using a directory separator of your choice (/ being the default), while the catfile stuff happens behind the scenes. You can just say:

my $path = path 'dir/subdir/file.txt';

What it does is it splits the path string on each occurrence of the forward slash and feeds the resulting list of path components to File::Spec->catfile, which reassembles them using the appropriate OS-specific directory separator, and constructs the OS-specific path that you want.

The module is up on Github, and should also be available on CPAN shortly.

]]>
]]>
Perldoc and I Would Like to Wish You a...tag:blogs.perl.org,2011:/users/michal_wojciechowski//954.26072011-12-24T12:48:14Z2011-12-24T12:51:55Zperldoc perlsyn | perl -ne 's/^.*?"|\\.*$//g,print if/1\.\.15/'...Michał Wojciechowskihttp://odyniec.net/perldoc perlsyn | perl -ne 's/^.*?"|\\.*$//g,print if/1\.\.15/'
]]>
Alternative Dancer Templating Enginestag:blogs.perl.org,2011:/users/michal_wojciechowski//954.25832011-12-18T13:59:47Z2011-12-18T21:56:26ZDancer uses a simple model of interfacing with templating engines (based on Dancer::Template::Abstract) and makes it very easy to add support for new engines. Thanks to this, if you're not happy with the default simple engine or with Template Toolkit,...Michał Wojciechowskihttp://odyniec.net/
Dancer uses a simple model of interfacing with templating engines (based on
Dancer::Template::Abstract) and makes it very easy to add support for new
engines. Thanks to this, if you're not happy with the default simple engine
or with Template Toolkit, there is now
a dozen different alternatives to choose from. Let's take a look at some of
them.

]]>
Dancer::Template::Tiny

Template::Tiny is a lightweight engine which reimplements a subset of
Template Toolkit features. As the name implies, it aims to accomplish this with
as little code as possible. If you're using just the basic functionality of
Template Toolkit, you should be able to switch to Template::Tiny without any
modifications to template files (and you can easily go back at any moment).

Dancer::Template::Tenjin

Tenjin is a very fast templating engine
with implementations for many languages -- including, of course, Perl. Its great
performance comes from the fact that it uses the underlying language's
constructs to process templates, instead of defining its own templating language
and having to parse it. Support for this engine in Dancer is provided by
Dancer::Template::Tenjin.

Dancer::Template::Haml

Haml, which stands for "HTML Abstraction Markup
Language", brings a fresh, different approach to templating. It aims at making
templates short, clean, and as easy to read as well-formatted source code.
Dancer::Template::Haml is a wrapper around Text::Haml and lets you use
Haml templates in Dancer applications.

More

There are many more interesting templating engines ready to be used with Dancer,
such as Mason (provided by
Dancer::Template::Mason) or Xslate
(Dancer::Template::Xslate). Do a CPAN or
MetaCPAN search for "dancer template" to get a list of
all the available engines, and choose the one that suits you best. In the true
spirit of Perl, there's more than one way to write a template!

]]>
Serving Files with Dancer::Plugin::DirectoryView and Dancer::Plugin::Auth::Htpasswdtag:blogs.perl.org,2011:/users/michal_wojciechowski//954.25732011-12-13T22:16:10Z2011-12-13T22:33:32ZA while ago I was converting a simple PHP website to Dancer, and moving it from being deployed on Apache to Starman. There wasn't a lot of code, so rewriting went quickly -- but, the site used a few specific...Michał Wojciechowskihttp://odyniec.net/
A while ago I was converting a simple PHP website to Dancer,
and moving it from being deployed on Apache to Starman.
There wasn't a lot of code,
so rewriting went quickly -- but,
the site used a few specific features of Apache,
namely directory indexes (courtesy of mod_autoindex) to allow user access to directories/files on the server,
and htpasswd files to password-protect some of those directories.

I could just deploy the new Dancer website on Apache and keep using those goodies,
but I thought that it would be nice if Dancer itself provided similar features.
So,
I created two plugins that do just that: Dancer::Plugin::DirectoryView and Dancer::Plugin::Auth::Htpasswd.
Let me now show you how to use them.

]]>
Directory Indexes

Let's say we have a files directory under public,
and we'd like to allow users to browse it and download files.
Enabling directory access is as simple as including the plugin in our Dancer application:

package MyWebApp;
...
use Dancer::Plugin::DirectoryView;

And updating the configuration file (config.yml) to tell the plugin which directory should be made available, and at which URL:

plugins:
DirectoryView:
url: /pub
root_dir: files

That's it -- now, if we launch our app and point the browser at the /pub URL, we'll see the contents of the directory:

Protecting Directories with Htpasswd Files

As you might have noticed on the screenshot, there's a secret directory under files.
It contains some super secret data that should only be available to
authorized users, so now we're going to protect it using a htpasswd file.

First, let's create the htpasswd file and an user, named "alice":

$ htpasswd -c htpasswd alice

Once it is created, we need to put the htpasswd file in a safe location outside of the public directory, so let's create a new directory passwd and store the file in there.

(If you're migrating from Apache and already have the htpasswd file, you just need to copy it to your Dancer application.)

In our Dancer application, we include the Auth::Htpasswd plugin:

package MyWebApp;
...
use Dancer::Plugin::Auth::Htpasswd;

Now, we need to update our configuration file and add settings for the plugin. We'll tell it to protect the /pub/secret path, and to use the htpasswd file we just created:

The realm parameter lets us set the text that will be shown to the user in the login window displayed by the browser.

Let's see if our protection works. We restart the application and try to access the /pub/secret/ URL:

Great, our confidential files are safe. Only when we log in as "Alice", we'll be able to access them:

This post was originally published as part of the 2011 Dancer Advent Calendar.]]>
GitHub-friendly README files with ExtUtils::MakeMaker and Module::Buildtag:blogs.perl.org,2011:/users/michal_wojciechowski//954.25102011-11-30T23:39:52Z2012-02-28T20:25:54ZGitHub is a great place to host open-source projects and expose them to a wide community of developers, so it's not surprising that more and more Perl modules are making it their home. One of the features of GitHub is...Michał Wojciechowskihttp://odyniec.net/GitHub is a great place to host open-source projects and expose them to a wide community of developers, so it's not surprising that more and more Perl modules are making it their home.

One of the features of GitHub is that it checks if a repository has a README file in its root directory, and displays it on the home page of the repository. This makes the README file a good place to introduce your project to the public.

GitHub also understands a number of markup languages, such as Markdown and Textile, and if the README file is in one of these formats, it will be transformed into nicely formatted HTML. One of the supported formats is POD, which means that the standard documentation of a Perl module can be used as its README file and serve as the repository's home page (much like on CPAN).

Module::Starter, which is Perl's recommended tool for building modules, does not create a GitHub-friendly README file -- instead, the README that it produces contains installation instructions (the "perl Makefile.PL; make..." mantra) and a couple links to module resources. This means that if you want to have a GitHub-friendly README file in your module, you need to either create it yourself, or tweak your build script a bit to have it generate it for you automatically.

This article wouldn't be particularly interesting if I told you to now go and make the README file yourself, would it? So let me show you how to do this automatically with ExtUtils::MakeMaker and Module::Build based modules (generated with Module::Starter). I will demonstrate how to create two README files: one being the POD version (named README.pod), the other one plain text (named just README).

ExtUtils::MakeMaker

Since ExtUtils::MakeMaker creates a Makefile with shell commands, you can tell it to generate the README files using two core Perl command-line utilities: perldoc (to generate POD from module's source) and pod2text (to convert POD into plain text). Extend Makefile.PL by adding the shell commands as the PREOP attribute of the dist target configuration:

Now, when you run perl Makefile.PL and make dist, the two README files will be created for you.

Don't worry if running make dist produces warnings that README and README.pod are missing -- it's no big deal, as the warnings will only be seen by you when making a distribution package, and not by the user building the module.

Module::Build

Module::Build defines a docs action, and it's the appropriate place for the code that builds the README files. Two modules that you can use for this purpose are Pod::Select and Pod::Readme. In your Build.PL file, create a subclass of Module::Build, and define a subroutine named ACTION_docs, similar to the one shown below:

You can now run perl Build.PL, and then ./Build docs, to build the README files.

Remember to add the Pod::Select and Pod::Readme modules to configure_requires and build_requires, as shown in the above example.

Installation instructions

Since these methods overwrite the original README file provided by Module::Starter, the installation instructions in it are also lost. It's a good practice to always include installation instructions, so go ahead and add an INSTALL file to your module's distribution files. It can be really simple and straight to the point: