Browsing Posts in OpenSource

Just a quick note to advise that the ASX RSS feed at http://noisymime.org/asx is currently not functional due to a change in the source data format. I am in the process of performing a rewrite on this now and should have it back up and running (Better and more robust than ever) within the next few days.

A few months ago I switched from using the standard mythfrontend to Boxee, a web enhanced version of the popular XBMC project. Now Boxee has a LOT of potential and the upcoming Boxee Box looks very promising, but its fantastic web capabilities are let down here in Australia as things such as Hulu and Netflix streaming are not available here.

What we do have though is a national broadcaster with a reasonably good online facility. The ABCs iView has been around for some time and has a really great selection of current programs available on it. Sounds like the perfect candidate for a Boxee app to me.

So with the help of Andy Botting and using Jeremy’s Vissers Python iView download app for initial guidance, I put together a Boxee app for watching iView programs fullscreen. For those wishing to try it out, just add the following repository within Boxee:http://noisymime.org/boxee

Its mostly feature complete although there are a few things that still need to be added. If you have any suggestions or find a bug either leave a comment or put in a ticket at http://code.google.com/p/xbmc-boxee-iview/issues/list (A Google code site by Andy that I am storing this project at)

So that’s the short version of the story. Along the way however there has been a few hiccups and I want to say something about what the ABC (and more recently the BBC) have done to ‘protect’ their content.

The ABC have enabled a function called SWF Verification on their RTMP stream. This is something that Adobe offer on top of their RTMP products despite the fact that they omitted it from the public RTMP spec. That wouldn’t be so bad, except that they are now going after open source products that implement this, threatening them with cease and desists. Going slightly technical for a minute, SWF Verification is NOT a real protection system. It does not encrypt the content being sent nor does it actually prevent people copying it. The system works by requesting a ‘ping’ every 60-90 seconds. If the player can’t provide the correct response (Which is made up of things such as the date and time and the phrase “Genuine Adobe Flash Player 001″) then the server stops the streaming. Hardly high tech stuff.

In my opinion the ABC has made a huge mistake in enabling this as it achieves nothing in stopping piracy or restricting the service to a certain platform and serves only to annoy and frustrate the audience. There is a patch available at http://code.google.com/p/xbmc-boxee-iview that allows Boxee to read these streams directly, however until such time as this is included in Boxee/XBMC mainline (Watch this space: http://trac.xbmc.org/ticket/8971) or the ABC come to their senses as disable this anti-feature, this Boxee app will use the flash interface instead (boo!)

So that’s it. Hope the app is useful to people and, as stated above, if there are any problems, please let me know.

[EDIT]
I should’ve mentioned this originally, Andy and I did actually contact the iView team at the ABC regarding SWF verification. They responded with the following:

Thanks for contacting us re-Boxee. We agree that it’s a great platform and ultimately appropriate for iView iteration. Currently we’re working out our priorities this year for our small iView team, in terms of extended content offerings, potential platforms and general enhancements to the site.

Just some background on our security settings. We have content agreements with various content owners (individuals, production companies, US TV networks etc) a number require additional security, such as SWF hashing. Our content owners also consider non-ABC rendering of that content as not in the spirit of those agreements.

We appreciate the effort you have put into the plug-in and your general interest in all things iView. So once we are on our way with our development schedule for “out of the browser” iView, we’ll hopefully be in a position to share our priorities a little more. We would like to keep in touch with you this year and if you have any questions or comments my direct email is ********@abc.net.au.

[STATUS]
The app is currently in a WORKING state. If you are experiencing any problems, please send me a copy of your Boxee log file and I will investigate the issue.

Despite risking the occasional dirty look from a certain type of linux/FOSS supporter, I quite happily run the (currently) non-free Skype client on my HTPC. I have a webcam sitting on top of the TV and works flawlessly for holding video chats with family and friends.

The problem I initially faced however, was that my HTPC is 100% controlled by a keyboard only. Unlike Windows, the linux version of Skype has no built in shortcut keys (user defined or otherwise) for basic tasks such as answering and terminating calls. This makes it virtually impossible to use out of the box. On the upside though, the client does have an API and some wonderful person out there has created a python interface layer for it, aptly named, Skype4Py.

A little while ago when I still had free time on weekends, I sat down and quickly hacked together a python script for answering and terminating calls, as well as maximising video chats from the command line. I then setup a few global shortcut keys within xfce to run the script with the appropriate arguments.

I won’t go into the nitty gritty of the script as it really is a hack in some places (Particularly the video maximising), but I thought I’d post it up in case it is of use to anyone else. I’ve called it mythSkype, simply because the primary function of the machine I run it on is MythTV, but it has no dependencies on Myth at all.

The depedencies are:

Python – Tested with 2.6, though 2.5 should work

Skype4Py – any version

xdotool – Only required for video maximising

To get the video maximising to work you’ll need to edit the file and set the screen_width and screen_height variables to match your resolution.
Make sure you have Skype running, then simply execute one of the following:

./mythSkype -a (Answer any ringing calls)

./mythSkype -e (End active calls)

./mythSkype -m (Maximise the current video)

The first time you run the script, you will get a prompt from Skype asking if you are happy with Skype4Py having access to the application. Obviously you must give your ascent or nothing will work.

Since my last post regarding the conversion of media from Channel 9’s Catch Up service, I have been in discussion with the company behind this technology, Hiro-Media. My concerns were primarily around their use of the open source xvid media codec and whilst I am not a contributor to xvid (and hence do not have any ownership under copyright), I believe it is still my right under the GPL to request a copy of the source code.

First off I want to thank Hiro-Media for their prompt and polite responses. It is clear that they take the issue of license violations very seriously. Granted, it would be somewhat hypocritical for a company specialising in DRM to not take copyright violations within their own company seriously, but it would not be the first time.

I initially asserted that, due to Hiro’s use (and presumed modification) of xvid code, that this software was considered a derivative and therefore bound in its entirety by the GPL. Hiro-Media denied this stating they use xvid in its original, unmodified state and hence Hiro is simply a user of rather than a derivative of xvid. This is a reasonable statement albeit one that is difficult to verify. I want to stress at this point that in my playing with the Hiro software I have NOT in anyway reverse engineered it nor have I attempted to decompile their binaries in any way.

In the end, the following points were revealed:

The Mac version of Hiro uses a (claimed) unmodified version of the Perian Quicktime component

The Windows version of Hiro currently on Channel 9’s website IS indeed modified, what Hiro-Media terms an ‘accidental internal QA’ version. They state that they have sent a new version to Channel 9 that corrects this. The xvid code they are using can be found at http://www.koepi.info/xvid.html

Neither version has included a GPL preamble within their EULA as required. Again, I am assured this is to be corrected ASAP.

I want to reiterate that Hiro-Media have been very cooperative about this and appear to have genuine concern. I am impressed by the Hiro system itself and whilst I am still not a fan of DRM in general, this is by far the best compromise I have seen to date. They just didn’t have a linux version.

This brings me to my final, slightly more negative point. On my last correspondence with Hiro-Media, they concluded with the following:

Finally, please note our deepest concerns as to any attempt to misuse our system, including the content incorporated into it, as seems to be evidenced in your website. Prima facia, such behavior consists a gross and fundamental breach of our license (which you have already reviewed). Any such misuse may cause our company, as well as any of our partners, vast damages.

I do not wish to label this a threat (though I admit to feeling somewhat threatened by it), but I do want to clear up a few things about what I have done. The statement alleges I have violated Hiro’s license (pot? kettle? black?) however this is something I vehemently disagree with. I have read the license very careful (Obviously as I went looking for the GPL) and the only relevant part is:

You agree that you will not modify, adapt or translate, or disassemble, decompile, reverse engineer or otherwise attempt to discover the source code of the Software.

Now I admit to being completely guilty of a single part of this, I have attempted to discover the source code. BUT (and this is a really big BUT), I have attempted this by emailing Hiro-Media and asking them for it, NOT by decompiling (or in any other way inspecting) the software. In my opinion, the inclusion of that specific part in their license also goes against the GPL as such restrictions are strictly forbidden by it.
But back to the point, I have not modified, translated, disassembled, decompiled or reverse engineered the Hiro software. Additionally, I do not believe I have adapted it either. It is still doing exactly the same thing as it was originally, that is taking an incoming video stream, modifying it and decoding it. Importantly, I do not modify any files in any way. What I have altered is how Quicktime uses the data returned by Hiro. All my solution does is (using official OSX/Quicktime APIs) divert the output to a file rather than to the screen. In essence I have not done anything different to the ‘Save As’ option found in Quicktime Pro, however not owning Quicktime Pro, I merely found another way of doing this.

So that’s my conclusion. I will reply to Hiro-Media with a link to this post asking whether they still take issue with what I have done and take things from there.
To the guys from Hiro if you are reading this, I didn’t do any of this to start trouble. All I wanted was a way to play these files on my linux HTPC, with or without ads. Thankyou.

About 6 months ago, Channel 9 launched their ‘Catch Up’ service. Basically this is their way of fighting piracy and allowing people to download Australian made TV shows to watch on their PC. Now, of course, no ‘old media’ service would possibly do this without the wonders of DRM. Channel 9 though, are taking a slightly different approach.

Instead of the normal style of DRM that prevents you copying the file, Channel 9 employs technology from a company called Hiro. Essentially you install the Hiro player, download the file and watch it. The player will insert unskippable ads throughout the video, supposedly even targeted at your demographic. Now this is actually a fairly neat system, Channel 9 actually encourage you to share the video files over bittorrent etc! The problem, as I’m sure you can guess, is that there’s no player for linux.

So, just to skip to the punchline, yes it IS possible to get these files working on free software (completely legally & without the watermark)! If you just want to know how to do it, jump to the end as I’m going to explain a bit of background first.

Hiro

The Hiro technology is interesting in that it isn’t simply some custom player. The files you download from Channel 9 are actually xvid encoded, albeit a bastard in-bred cousin of what xvid should be. If you simply download the file and play it with vlc or mplayer, it will run, however you will get a nasty watermark over the top of the video and it will almost certainly crash about 30s in when it hits the first advertising blob. There is also some trickiness going on with the audio as, even if you can get the video to keep playing, the audio will jump back to the beginning at this point. Of course, the watermark isn’t just something that’s placed over the top in post-processing like a subtitle, its in the video data itself. To remove it you actually need to filter the video to modify the area covered by the watermark to darken/lighten the pixels affected. Sounds crazy and a tremendous amount of work right? Well thankfully its already been done, by Hiro themselves.

When you install Hiro, you don’t actually install a media player, you install either a DirectShow filter or a Quicktime component depending on your platform. This has the advantage that you can use reasonably standard software to play the files. Its still not much help for linux though.

Before I get onto how to create a ‘normal’ xvid file, I just want to mention something I think should be a concern for free software advocates. As you might know, xvid is an open codec, both for encoding and decoding. Due to the limitations of Quicktime and Windows Media Player, Hiro needs to include an xvid decoder as part of their filter. I’m sure its no surprise to anyone though that they have failed to release any code for this filter, despite it being based off a GPL’d work. IA(definitely)NAL, but I suspect there’s probably some dodginess going on here.

Using Catchup with free software

Very basically, the trick to getting the video working is that it needs to be passed through the filter provided by Hiro. I tried a number of methods to get the files converted for use with mplayer or vlc and in the end, unfortunately, I found that I needed to be using either Windows or OSX to get it done. Smarter minds than mine might be able to get the DirectShow filter (HiroTransform.ax) working with mplayer in a similar manner to how CoreAVC on linux works, but I had no luck.

But, if you have access to OSX, here’s how to do it:

Download and install the Hiro software for Mac. You don’t need to register or anything, in fact, you can delete the application the moment you finish the install. All you need is the Quicktime component it added.

Grab any file from the Catch Up Service (http://video.ninemsn.com.au/catchuptv). I’ve tested this with Underbelly, but all videos should work.

Voila. Output will be a file called ‘<filename>.clean.MP4′ and should be playable in both VLC and mplayer

Distribution

So, I’m the first to admit that the above is a right royal pain to do, particularly the whole requiring OSX part. To save everyone the hassle though, I believe its possible to simply distribute the modified file. Now again, IANAL, but I’ve gone over the Channel 9 website with a fine tooth comb and can see nothing that forbids me from distributing this newly encoded file. I agreed to no EULA when I downloaded the original video and their site even has the following on it:

You can share the episode with your friends and watch it as many times as you like – online or offline – with no limitations

That whole ‘no limitations’ part is the bit I like. Not only have Channel 9 given me permission to distribute the file, they’ve given it to me unrestricted. I’ve not broken any locks and in fact have really only used the software provided by Channel 9 and a standard transcoding package.

This being the case, I am considering releasing modified versions of Channel 9’s files over bittorrent. I’d love to hear people’s opinions about this before doing so though in case they know more than I (not a hard thing) about such matters.

After my post yesterday about the ‘rallyduino’ a few people emailed me asking for the source, so I threw everything together quickly into a Google Code hosted project. I did want to get it into SVN anyway as having only local copies was starting to get me nervous.

The power regulator I’d ordered arrived last night (Turned out to be well and truly overkill, but it only cost $10) so now just awaiting the Arduino prototyping shields and I can give this a real worlds test.

So recently I’ve started playing slightly more seriously with my Arduino, I’ve even gone so far as having a plan on what the hell to do with it. For some time I’ve wanted to try my hand at putting together a homemade Rally computer and the Arduino seems to almost be the perfect platform for this (Though not without one or two shortcomings). For those not familiar with them, a rally computer in its most basic form is essentially just a glorified odometer, in other words, it measures the distance your car has gone. Very briefly the differences are:

Considerably more accurate

Can count up and down as well as being able to pause

Resetable, usually via a remote control

Usually also calculate average speed

If you want to lookup existing rally computers, I recommend you checkout manufacturer websites such as Brantz and Terratrip.

So why make your own when there’s already such good existing products? Well I can give you 3 reasons: price, openess and fun. All the units currently on the market are quite expensive ($500+) and are all very much proprietory, closed systems. My goal is to put something together than costs about $200 and is as open as possible.

Do demo what I’ve got going so far, checkout the video below:

So as you can see, the system is up and running with all the basic functionality. I need a few extra pieces of hardware (Case, power regulator and prototyping shield), before I give it a test in a real world scenario, but at the moment I’m confident things should work OK. The biggest problem I’m having at the moment is the responsiveness of the Wii nunchuck as a controller. It works OK in tests, but when you get the Arduino processing the input pulses (At around 150Hz) things start to slow down. Obviously priority is given in the following order:
Pulse Counting->Speed/Distances Calculations->Screen Updates->Nunchuck input

That’s about all for now. I’ll probably post about this again after I’ve done a real world test on a car so it will have to wait until the extra bits and pieces arive. The source code isn’t yet available (Simply because I don’t think there’d be much interest), but if anyone would like to see it, just leave a comment and let me know.

I know I’ve been promising to do another screencast, specifically showing the music player in Gloss, but I’m afraid it isn’t done. What is done however is Gloss 0.1, including a new screencast showing all its features!

Whilst being very much a developer / testing release, it is moderately stable and works ‘out of the box’ on multiple different systems I’ve been testing on. It is available for download at http://gloss-mc.googlecode.com/files/gloss_0.1.tar.gz or if you’re on a debian / ubuntu based system, an apt package is available in my launchpad PPA: https://launchpad.net/~gloss-development/+archive. If you want to check it out before downloading, a screencast is below (Quality warning: Where is the killer linux screencast app?):

So what’s actually been done? A LOT of work getting what was in there already to a stable, portable state. That’s not to say it won’t crash, some areas are still quite fragile, but everything that’s there will work given the right setup. The new Gloxygen theme has been included also. This initially started life based around the KDE Oxygen iconset, but has evolved a little bit since then. It has the advantage of being fully open licensed, unlike the Pear theme which is of questionable origin (For this reason it is not included in the apt package above).

The plan going forward is to get as much testing as possible done ASAP with the hope of having a 0.1.1 and maybe a 0.1.2 release out the door quickly. When Clutter 0.8 is released, SVN will temporarily break as the API is updated accordingly, meaning that the ubuntu Hardy package will also break given that there is no Clutter 0.8 available for it.

That’s probably about all for now. I urge you to have a play with the release above if you have a chance and log any problems you find (And you will find them) in the bug tracker!

I’ve read a few blogs posts recently regarding consumer NAS devices, specifically linux based ones (See entries from Peter Hardy and Matt Bottrell) and, for the most part, these have been fairly positive. So as a warning to others, I thought I’d pop up my experience.

A few weeks back I got a call from my Dad saying he’d bought a network HDD and wanted to know if i could come around and set it up for him. No problems, I thought, should just be a matter of plugging it into his (fairly basic) network and setting up mappings on the various machines he runs.

I head around and find a Western Digital ‘My Book World Edition’ 500gb drive. It came in some fairly slick packaging (Very Apple) and generally looked pretty good. The hardware part of the install was a breeze and within minutes I was installing the ‘MioNet’ software on his main machine. My instincts had told me that installing software for a device like this shouldn’t be necesary, but one of the things it offered was secure remote access via a browser, which could come in handy at times (my dad travels a bit and is forever asking me to pop around and email him files he forgot to put on his laptop). This is where the problems started. Upon installation and reboot of the machine, the network interface was completely gone. Nothing, nada, dead. I tried everything I could think of for the next hour or so but nothing I did would bring it back. Finally I uninstalled the software, rebooted, and everything was fine again. Not the most reassuring experience.
Nevertheless, I continued, Dad can live without the crummy software as the device offers a direct SMB share ability. I setup a few shares via the devices web interface and mapped them on 2 machines. Everything seemed fine. So we started the initial backup (About 30gb of data) but almost immediately noticed 2 things:

The speed. At its best, this thing got up to about 4mb/s, pretty poor for something that was on a gigabit network (It is a gigabit interface on the hdd).

After about 1gb of transfer, the copying terminated saying that we no longer had permission to write to the share. I tried creating a text file. No go. In fact, it wasn’t until we rebooted the unit (Takes about 90secs) that we again had write access… For about 2 minutes when it again cancelled the transfer and refused to let us make any changes to the drive.

Being somewhat stumped, I called Western Digital to see if we had a dud unit. I eventually got onto a rather stressed sounding guy who advised me that this was a known issue, to upgrade my firmware and cross my fingers (ie there was no promise the upgrade would solve the problem). He also advised that the 4mb/s was about the theoretical maximum throughput of the device due to a hardware restriction (The speed of its CPU). Now, putting the firmware issue to one side for a second, how can you knowingly advertise a device as having a 1000mb/s network interface when the rest of the hardware struggles to cope with 100mb/s speeds!?! I wanted to unload on this guy, but deep down I knew he’d heard it before and that a return of the drive was already looking likely.

Getting back to the firmware upgrade. This did eventually go through (Was about an 80mb download) however it downright refused to do this until I had blanked the drive. It continually advised that there were current connections (Despite me disconnecting all shares, rebooting PC/Device, plugging a laptop directly into the unit etc) and refused to do the upgrade with these in place. Long story short, the firmware upgrade made absolutely no difference. The problem having write access still occurs for transfers over about 800mb making proper backups all but impossible.

I was almost disapointed then to discover that this device actually runs linux. Presumably the Windows sharing is provided by Samba which makes me really wonder what is causing the access problems. The unit is obviously under powered hardware wise based on its transfer rates, so I’d like to think it is dubious quality hardware causing the other problems as well. Whilst I could delve in and start tinkering to try and fix things, this is a consumer device, that was definitely not purchased to be a tinker-toy, and I didn’t want to unkowingly change something that could potentially lead to data loss (now or down the track).
The result of this ordeal is that the unit will be going back shortly, it is simply unusable in its current state, and all I can really say to other people is to avoid this device at all costs. It really is a poorly designed package (HW and SW) that will cause you move grief than benefit.

One of the things I’ve been constantly delaying on Gloss is the development of the music player module. For a long time this was simply because I had no idea in my own head about what it should look like. I’ve bitched and moaned along with all the others about the existing MythMusic interface, but when it was all said and done, I didn’t have anything better to offer.

Well that is now a thing of the past. There’s a plan in my head and some code in SVN. What I’m picturing is a mix of the interfaces from Elisa and FrontRow, hopefully being a little simpler in the end. One of the crucial things I decided early on with this idea though was that it would require images. Lots and lots of images. Nobody likes sorting through a list of band names when you could be rolling through some form of pretty image display (ie Coverflow).

The obvious problem with this though is, of course, actually having the images. ID3 data in the files themselves is ok, but it really doesn’t represent the band itself. If you’ve got a band with 5 albums, which album cover do you pick to represent the band? So whilst Gloss will now pull images out of ID3’s, the solution is for the higher level pics fairly obvious, Last.fm. Last.fm offer a reasonable web service interface allowing you to get information, including images if you look hard enough, on bands, albums, songs etc. Integrating with Last.fm wasn’t too difficult, although bringing it all together is still a work in progress.
Currently, Gloss will run through your existing music collection and download (in the background) images for artists and albums, storing them locally for future use. This is a fully automated process and has about a 95% success rate. Once the images are downloaded, artists and albums can be searched in a pretty growing/shrinking image row, similar to what Elisa currently uses. I’m thinking in future that I will also pull general info about bands (description, tour dates etc) from Last.fm, but these would obviously require further DB updates, which I am hesitant to do.

So that’s about it for shiny new stuff type updates. There’s been a bit of work ‘generifying’ some code to make it usable across the whole system rather than just in specific modules. Code reuse is a glorious thing when you sit down and make things useful
Depending on how busy I am over the next week, I’d like to have another screencast with the basic Music interface up within 2 weeks, but there’s still a lot to do.