A much better interface could be provided by treemaps. Ideally allowing the user to drill-down into deeper levels of the package namespace hierarchy. (It needn’t be this flashy, just 2D would be fine :-)

In case you’re not familiar with them, treemaps are a great way to visualise hierarchical data. Here’s an example treemap of the disk space used by the files in a directory tree (from schoschie)

Perl already has a Treemap module, which can generate treemap images via the Imager module. Treemap is designed to support more output formats by sub-classing.

I guess it wouldn’t be hard to write a sub-class to generate HTML client-side image map data along with the image, so clicks on the image could be used to drill-down into treemaps that have more detail of the specific area that was clicked on.

More interesting, and more flexible, would be a sub-class to generate the treemap as a Scalable Vector Graphics diagram using the SVG module (and others).

I’m not going to be able to work on either of those ideas anytime soon.

29 thoughts on “Generate Treemaps for HTML from Perl, please.”

Flare (http://flare.prefuse.org/) is an open source Flex/ActionScript toolkit for data visualizations, including treemaps. You get SWF instead of SVG, it that good enough? I’ve never hooked up ActionScript to perl, but I’m sure it has been done.

Flare looks nice, and would enable good visualizations for other NYTProf data (e.g., calls between packages/subs could be visualized like http://flare.prefuse.org/apps/dependency_graph). I’ve no experience with Flash (or Flex/ActionScript) so I’ve no idea how I’d get the perl code to integrate with Flex/ActionScript and generate the .swf file. (I’d welcome details of how to make it all fit together in a simple, reliable, and portable way.)

SVG has the advantage of simplicity, not to mention being an open standard.

Tim, if you’d made it to OSS Bar Camp a few weeks back ;) you might have been at the talk where I learned that Adobe’s Flex Develop is now an entirely open source platform; you can dl the tools for free, it seems. However I’m pretty sure it’s entirely Windows-only.

What you could do is write a Flash app that reads an XML file describing the treemap to display, and displays it, linking back out to whatever other pages it needs and/or parsing other XML files if necessary. in other words, some kind of Treemap Displayer flash widget. sounds like a fair bit of work though.

Of course there are a lot of jobs for Perl, just as there were for COBOL after it became defunct. You can hide your head in the sand if you wish, but from what I’ve seen you’re quite a good programmer. You should consider putting some time in with a modern language so that when the legacy jobs dry up you’ll have somewhere to jump.

How do you envisage the drilldown working? Do you show one level of calls/statements per page, with each call drilling down to the next level of calls below. I’ve generated a simple Treemap::Output::SVG module that does the same as the equivalent Treemap::Output::Imager module. It doesn’t do drill down yet, and the nature of the Treemap module means that all the blocks are sized in the Treemap module itself, which (presumably) wouldn’t work well if you want to drill down through a call to a full sized page.

How did you envisage getting input into the Treemap? A Treemap::Input::Devel::NYTProf module, via the Treemap::Input::XML module or some other way?

If you drop me an email I can send you the Treemap::Output::SVG module source, it’s not on CPAN as it’s only a proof of concept at the moment.

Dave, using separate pages for each possible drill-down view would be very wasteful and a poor user experience. The drill-down really needs to be done in the browser e.g., via JavaScript. For example, see http://js-treemap.sourceforge.net/tests/demo.xml.html (I thought it didn’t have drill-down, but it does: click on the underlined words).

I see your point about the blocks being pre-sized, making drill-down on the client a tricky problem. Umm. Though I guess it’s only really a problem if you want the drilled-into subsection of the tree to be ‘fully fitted’ into the viewing frame. We don’t really need that. So here’s an idea: perform drill-down by effectively SVGZoomAndPan’ing or SVGFitToViewBox’ing such that the section of the treemap that was clicked on now fills either the width or height of the viewport, whichever comes first. (I say ‘effectively’ because I’ve only spent 5 minutes skimming the SVG spec to find some straws for me to clutch! There’s quite possibly a better way.)

Re getting the data into the treemap, I’d make Treemap::Output::SVG an optional dependency for Devel::NYTProf. If it’s installed then the nytprofhtml script would automatically use it to generate a treemap svg file and write a modified index page that would refer to the svg.

Tim,
Would it be easier to use the JS from http://js-treemap.sourceforge.net directly in a page generated by nytprof? It dynamically scales the treemap to the current browser window, something that the Treemap set of modules doesn’t do.

The examples take JSON or XML, and the JSON version doesn’t appear to require you precomputing parent sizes based on their children, something you’d need to do in any Treemap::Input module you wrote (Treemap needs a Treemap::Input::[Dir|XML|…] object to manipulate).

Hi Dave. Yes, http://js-treemap.sourceforge.net/ looks appealing now I know it supports drill-down. On the other hand, the project seems to be dead. I’m trying to contact the author. Re dynamic scaling for Treemap modules, I figured that would be easy enough to add on the browser for the *Scalable* Vector Graphics support.

Hi Tim, I’ve put a mock-up of an SVG drilldown at http://79.170.40.10/sentrasystems.co.uk/testing/test_zoom.svg (the domain is in the middle of being registered – after that it will just be http://www.sentrasystems.co.uk). It only works in Firefox 3+ at the moment, IE has issues with SVG DOM support that I’d need to resolve further – it uses a no longer supported SVG plugin from Adobe (amongst many other things!). To drilldown – click on a box, to return click on the surrounding box. Links don’t have underlines at the moment as FF doesn’t implement the text-decoration attribute yet.

I wonder what’s going on! Firefox 3.0.10 on win xp, mac and linux all allow me to drilldown by clicking anywhere within the box. If it shows the whole treemap it normally means the javascript has broken (it makes the elements invisible in the init(evt) function). Are you running Firebug on Firefox – if so could you step through the Javascript and let me know where it breaks.

I still can’t see the whole treemap in either FF or Safari unless I disable Javascript.

I think font fitting could be dropped. It tends to distract from the boxes and there’s no great need for it. Just use a small but readable font. The names need to be written into the top label area (an area that doesn’t represent a vaue) so they’re readable and not covered by inner boxes.

A treemap for NYTProf would be extremely useful. Unfortunately I don’t have the time for an elegant solution. For Mac users a interesting quick and dirty solution would be to use MacFUSE to map NYTProf results into a file system representation and have Disk Inventory X create the treemap.

I’ve code in NYTProf now generating treemaps using js-treemap (and before that using JIT). It’s not perfect but it’s also useful and not much work. The main work is in munging the data structure to fit the needs t the treemap code. (Which is why I stopped using JIT as the data structure is horrible.) It’s not too much work to switch to other treemap renderers so I may switch again in future (e.g., perhaps to Dave’s SVG based treemap when it’s at least as good as js-treemap).

I subscribed to the post but didn’t get an email on posts after 27th May. I was looking at JIT in nytprof, mainly because it was an already supported toolkit and that worked cross-browser. Any SVG implementation has an issue with IE and the now unsupported ASV plugin from Adobe. If it makes sense to continue with a statically generated treemap using SVG, can you send me a copy of the new nytprof so that I get a better idea of what you’d want from the final package.

Short answer is hang back for now. I’m running with js-treemap at the moment. Hopefully once the dust settles the code in nytprofhtml will get refactored to make it easy to switch to alternate visualizations. (I have some others in mind that would be easier with JIT but the data format drove me nuts. I’ll probably revisit it sometime.) Thanks for exploring this Dave.