Software Engineering, and the things I do to cope.

Menu

If you find yourself developing a distributed system, it’s very useful to have a logging subsystem which allows you to see output from multiple nodes in one place.

Python’s Logging module is quite comprehensive, it gives a lot of ways to format messages and a good level of control over what happens. For any given Logger object, there can be multiple Handlers which control where log messages go.

For remote logging Python provides two handlers StreamHandler and DatagramHandler, for TCP and UPD connections respectively. This example focuses on using the DatagramHandler for a UDP connection.

Step 1: Adding the Handler

This part is easy, if you have a Logger object already instantiated then get the DatagramHandler class from logging.handlers and create an instance with the parameters for the IP address and the port of the remote listener.

from logging.handlers import DatagramHandler
ip_address = "127.0.0.1" # Localhost, for demonstration
port = 5000 # Any port that is not in use
udp_handler = DatagramHandler(ip_adress, port) # Create the handler
log.addHandler(udp_handler) # Add the handler to the logger

So that bit’s easy enough! Note that no formatting information is needed at this point, that is handled by the receiver.

Step 2: Receiving the message

This is the part that gets difficult. The receiver will get the message, but rather than being a nice formatted string, it will receive a serialised (In Python terminology: pickled) object of type dict (A Python dictionary). This contains all message parameters we need to format the output.

First: we’re going to make use of four modules: socket, struct, cPickle, and logging.

Second: a socket is needed to receive information sent to it. We create the socket for use with IPv4 addresses and with intent to use UDP as shown. When the socket is bound using the bind() method, we can start accepting connections.

Third: We need a Logger object to work with the LogRecord object we will decode. At this point any formatting rules can be used. In this example, a simple StreamHandler is used to print messages to StdOut.

The socket waits on the recv() method until data is sent to it, after which the socket tries to read 2KB of data from the stream. The stream of data contains two things: the length of the data packet (first 4 bytes) and the pickled information. Using the struct module, the first four bytes are translated into an integer, then the cPickle module is used to unpickle that data, from this we get the dictionary. The dictionary is simply turned into a LogRecord object by the logging module, then the Logger object can handle it like any other message.

As a note: if you expect datagrams to ever be longer than 2kb, you will need to use a loop to ensure you receive all data else you will fail to unpickle the data.

The recv() method takes data from the socket, but if the address of the other client is needed, use recvfrom() instead, which returns a tuple containing the data and the address of the client. If more information is needed, consider the following…

Step 3: Sending Additional Data (Optional)

Sometimes, certainly with distributed systems, it’s important to be able to show identifying information in the log. This could be placed in every single log message individually but that’s tedious and may result in a lot of unnecessary code. My solution was to add a dictionary of static information to the handler which is put in ever record it handles.

Attributes of the LogRecord object are kept in its namespace, overriding the handle method in this way allows us to put all our attributes in the record before it’s pickled and sent. An AttributeError will be raised if you inadvertently attempt to overwrite any of these existing attributes.

On the other end, a formatter can be used to display the information. Say we create an instance of our CustomUDPHandler with the parameters: (‘127.0.0.1’, 3000, {‘ident’: “Client #5”}) then the attribute ident will be a part of the record sent to the other side. Access it as such:

I’ll be blunt – I am an old man. I take joy in disrupting traffic at zebra crossings; I complain about popular culture; I complain about politics; I complain about buses; I complain about traffic; I complain about people who complain about things; I complain about people who complain about people complaining; my hearing isn’t very good and I’ve got the memory of a BBC Micro. The mere fact I’m actually 21 has no bearing on the fact that my 65+ free bus pass is already in the post. That’s not to say that I’m out of touch with the people of my generation but rather that I’m atypical of most 21-year-olds from the North East. Many of my interests do tend to be quite niche, like ale and genuinely enjoying working with databases.

I noticed on my Facebook feed today that Cullercoats Brewery (Great little brewery in Wallsend, not Cullercoats) had linked to this article on Big Mike’s Real Ale blog. It’s a good read with some well formed opinions, I recommend you give it a read before continuing with this post as to give it context.

Real Ale is a true British success story. Given the market dominance of cheap fizzy lagers all those years ago and poor cask ale, the cask market’s recent overtaking keg as the preferred dispensing method combined with the fact that there are now over 1000 active licensed breweries in the UK, show that the beer market is stronger than ever – and certainly not the secondary choice. Although cask ale has proven itself a champion in the market in general, it still struggles to make itself popular for the under 25s. (Or even the under 30s)

One of the reasons that’s often touted as the explanation for why young people stay away from ale is because it’s “an old man’s drink”. As Big Mike states in his post, we see our parents and grand parents drinking it and want to be different. I don’t think that quite encapsulates the point. It’s not that young people see their elders drinking real ale, it’s the image of real ale – old men with flat caps in a pub with a less than appealing smell. And in some places, this is the case but most pubs have shed this stereotype of ale drinkers. I don’t even own a flat cap but then again I’m not from Yorkshire.

I think what’s more key to the reason for young peoples aversion to ale is peer pressure. When I started drinking with my friends, I drank lager – not because I thought ale was for smelly old men but because that’s what my friends drank. Lager was easy enough to get drunk on (which, lets be honest, is why young people drink) and it was easy to get. It’s also a lot simpler, nowadays when I walk into a pub it can take me a while to decide which beer I want to drink (Much to the distress of the bartender) whereas lager is lager is lager. A pint of lager – done. Also helps that, unlike ale, you can easily buy lager in bulk. Then again you can buy multi-packs of beers such as Speckled Hen or Old Peculiar but I’m not personally going anywhere near that hangover juice. To some, the fringe of the younger generation, individuality is paramount but ultimately we, as humans, are social creatures so it’s very much a case of trying to fit in with ones peers – even for the fringe.

I still believe image plays a big part of the problem. Lager is seen as cool, it’s the “in thing” at the moment, the adverts are catchy and the logos are flash. Have a wander over to the Pumpclip Parade blog, it’s ok I’ll wait for you to finish. You’ll notice that these pump clips are absolutely abominable. These are real pump clips seen in pubs used to advertise beer. I have to say, the outright worst offender is Northumberland Brewery – it doesn’t help that their beer is usually rubbish too. Marketing *spits on floor* isn’t just about making people aware of a brand, it needs to entice and make the product seem desirable and most marketing of ale fails to do this. Now take heed here, this does not mean that the correct solution to getting young people into beer is to “be down wiv da yoof” because if I see anybody doing that I will destroy them. What we, the beer drinking community, need to do is make young people excited about beer. Show young people that ale is amazing.

The two years I’ve spent at university so far have easily been the best years of my life. I’ve met loads of new people, lots of new friends, and I’ve become part of a group I hold dear. I was a member of, and then secretary of, the Real Ale & Cider Society at Northumbria Students’ Union. I cite this because each year we’ve grown in size as a society; only a few of the new members we get join us because they like ale whereas most just want to avoid the typical nightclubs and vodka trebles rubbish that pollutes most student culture. But just by being with our society, we show them that ale isn’t about smelly old farts, that it’s really a fascinating area of British culture which is so diverse! A couple of members of our society actually home brew which is pretty awesome. Our society has gotten people excited about beer!

That doesn’t mean we’re always welcome. I know I’ve been to a few places with friends where the clientèle give you funny looks, or even death stares, for the heinous crime of daring to step into their territory. There’s the prejudgement that all young people are loud, noisy and disruptive and obviously all young people shouldn’t be tarred by the same brush. Though, to be honest, I’ve rarely seen this outside of rural “local pubs for local people”. One exception being a small pub in Hartlepool – it’s nice and so is the beer but the owner seems averse to the under 25s as though they carried bubonic plague. Most places I’ve been to in Newcastle have usually been welcoming to us, as a result we go back there regularly and drink more beer. Everyone’s happy!

The scene itself is becoming more modern and, much as they may be tutted at, a lot of newer breweries trying experimental things like super-duper hoppy beers are attracting more attention from the younger crowd. Some of the excellent offerings I’ve seen from new breweries such as Tiny Rebel, Tyne Bank and Summer Wine Brewery (to name a few of a large group) are the exciting frontier of beer that’s going to entice and prevent stagnation of a crucial aspect of British culture. That’s right, what will save British culture is not banishing brown people but making good beer! Yeah, I went there!

I think the end points that Mike makes are sound. If you want a diverse customer base for beer, that is you want a wide variety of ales, then you need to be ensuring that you have a wide variety of people to enjoy these beers. And it’s not about simply educating the young as though there’s another class we’re missing, just after PE we’ve got beer tasting, but doing more to make cask ale seem a viable option at the bar. Offer tasters to customers, be active in trying to make the beer exciting and young people will be interested! Get psyched!

I realise it has been absolutely ages since I’ve posted anything to this blog. Life’s been rather hectic as of late and I’ve not had the opportunity to write anything here, which is a shame because my life has taken a rare turn for the interesting as of late – as I’m writing this I’m currently staying in a hostel in France although by the end of next week I should be living in Geneva, Switzerland.

Why am I here? Well I’ve been quite lucky, I’ve been accepted onto a technical placement at CERN. Yes, CERN! It’s an amazing opportunity, not only to experience a year living in mainland Europe but also a technical placement at one of the most prestigious research institutions in the world. I don’t actually start until Monday so there isn’t much to say until I actually start, other than I’ll be working on the Inspire project as a Python developer – in short, it’s a really big database!

I set off from Newcastle at mid-day towards Manchester Airport. The four of us (myself, Dan, James and Jake) who are also working at CERN, being computer science students, spent a decent amount of time on the trip on laptops. Got a bit bored of that (No internet) so we moved onto playing cards – an interesting game was had.

Traversing Manchester Airport was a bit of a pain, but also rather insightful. Managed to get myself a pint of Thornbridge Kipling from a bar in the departure lounge which came up to £4.60. Ouch! What also doesn’t help is, not being particularly fond of planes anyway, I spent most of the day (and a few days before) suffering crippling anxiety – not fun!

Getting into Geneva was a relief, once I was off the place I managed to start feeling the excitement. What I’ve discovered since then is that Switzerland is a country where zero fucks are given. Not a single one. Walking out the airport to wait for the bus a car pulled up and a man got out, as this blocked the road the car behind honked it’s horn – the man getting out just shushed them. A mix of serenity and arrogance – nice.

The past few days house-hunting have been immensely stressful. The property market for lets in Geneva is not good! We initially wanted a four bedroom house, furnished, in Meyrin – oh how our standards have changed! We’ve settled for a four-bedroom flat in central Geneva which is quite pricey but hopefully worth it. It’s about 20 minutes from CERN so not a bad commute either!

Switzerland itself is beautiful, of what I’ve seen so far. Mountains cover most of the horizon, although the past few days have been rather overcast so the cloud has covered them. I’ll get some nice pictures to show you all.

I’ve had a touristy sort of a day today to investigate central Geneva, we got to see Lake Geneva and the big jet. Also stumbled upon a big protest, not sure quite what it was about but they certainly seemed to annoy the drivers, one of the protesters started smacking an encroaching car with their Solidarity flag.

I’m just chilling in the French hostel at the moment. Begin work on Monday – it’s food time!

I take the Star Trek philosophy of life – the one that says being human is inherently a little bit shitty. It’s not the worst thing in the world at all, being a male Anglerfish is probably a lot worse since they… well lets not get into that – the point is that we, as a society, have brought pollution, war, poverty and Piers Morgan into existence. Most of us recognise these as bad things and rightly so, a lot of suffering has been caused by Piers Morgan as well as a lot more by war, poverty and pollution to name just a few dangers to the world.

But that’s what gives hope to the lifeless granite-stone I call my heart. That there are people who passionately care about these issues, people who are willing to stand up for what they believe – people who protest these abominations and people who punch Piers Morgan in the face. But that’s the part of Star Trek philosophy I like! I love the idea that we, as people and as a society, are able to better ourselves. We can use our scientific understanding of the universe to make peoples lives better, we’re able to use it to grow enough crops to feed the planet, to clean drinking water and to fly Piers Morgan to another country so I don’t have to be reminded every day that he exists.

And thus comes the New Year. After we’ve recovered from National Hangover day, we task ourselves with improvement. Either we want to lose weight, learn a language, tidy ourselves up – things that we feel make ourselves better. And then the inevitable disappointment ensues. One of the problems we have keeping our new years resolutions seems to be a lack of motivation. The purpose of the new year is “new beginnings” but we quickly lose sight of that when we realise that we’re much happier on the sofa with Friends and a packet of Minstrels than we are waking up early for exercise and tofu.

Maybe our motivation is wrong. I haven’t made New Years Resolutions since I was a teenager, partly because I don’t see the new year as a significant mark in my own life. I still celebrate the new year but that’s mostly because Talisker is gorgeous. I consider my birthday the point at which I think of myself as “new” in the year and that’s when I make resolutions to myself to be a better person and for the most part I feel I’ve been successful.

The dissapointment to me comes a few days after when everybody on Facebook collectively gives up.

I’ve often heard it said that the people who are most critical of us are ourselves. For some cock-sure people this obviously isn’t true, however for the majority of normal people who acknowledge their own flaws and shortcomings – motivate yourself! Jog on… and pass me my shot glass – I need another drink!

Citrix Receiver is a useful cross-platform tool for accessing programs and files on other networks. My university uses Citrix for their desktopAnywhere service but only provides support for MacOS and Windows which is a pain because installing Citrix on Ubuntu is not quite smooth sailing. So here’s a quick guide to cover install and all the hurdles I went through trying to set it up.

1. Download your packages

You can find all the files for Citrix Receiver on the Citrix website in various different flavours. 64 bit packages are at the bottom of the page. Be aware of the file type you’re downloading; Ubuntu users will want to download the .deb files. You can also download the USB Support Package which gives Citrix the ability to read and write from your USB flash drives during a session.

Note: If you’re unsure about which package you need, x86 (32 bit) or x86_64 (64 bit), then run the command uname -i in terminal and you’ll see the arch type of your machine.

OpenMotif is required for the graphical configuration menu of Citrix Receiver but not required for general use so we won’t bother with it for the time being.

2. Installing your packages

Due to a few prompts that may pop up during installation, it’d be best to install through the command line. There’s also a bug that prevents successful installation on 64 bit systems so there’s two ways to go about this.

32 bit: Open the terminal and find where you downloaded the files, for me that was Downloads, and install with dpkg as such:

~$ cd Downloads
~/Downloads$ sudo dpkg -i icaclient-12.1.0_i386.deb

Use the same command to install the USB support package if you wish to do so.

~$ sudo dpkg -i ctxusb-2.2.0_i386.deb

64 bit: Open the terminal and find where you downloaded the files, for me that was Downloads, and we want to partially install with dpkg as such:

Open your favourite text editor (gedit will do) and find icaclient.postinst in the directory /var/lib/dpkg/info/

~$ sudo gedit /var/lib/dpkg/info/icaclient.postinst

It’s a bit of a long file but you want to find the text near line 2648 that reads

echo $Arch|grep "i[0-9]86" >/dev/null

Replace that line with this

echo $Arch|grep -E "i[0-9]86|x86_64" >/dev/null

Save the file and finally, run this command to configure the package.

~$ sudo dpkg --configure icaclient

ICA client should be installed. Should you wish, you can also take this opportunity to install USB support in the same way by using dpkg -i <filename here>.

3. Supplying authentication certificates.

The Citrix Client does not supply many of it’s own SSL certificates for secure connection authentication, fortunately for Ubuntu users – Firefox does! You can use the following command to copy them over.

Any specific CA certificates that your Citrix admin requires can be placed in that folder too. If you get the error You have not chosen to trust “<name here>“, the issuer of the server’s security certificate. then you’ll need to place the appropriate CA certificate in the /usr/lib/ICAClient/keystore/cacerts/ directory.

4. Give it a go!

I’d had a few issues installing the Citrix Receiver onto 64 bit Ubuntu but following those worked for me and… voilà!

In the online age of uncertain digital rights, it’s crucial for individuals to take a stand and raise awareness of the issues that affect all of us online. There’s several ways within the online world to demonstrate and raise awareness of causes such as e-petitions or social networking groups, but one of the more controversial methods is hacktivism. The term hacktivism is a portmanteau of the wordshacker and activist which tends to refer to the use of unauthorised computer access to further an agenda, usually political or social. Hacktivism itself can be a fairly dubious issue, for all the power it can grant hackers of varying shades of grey, it could potentially be an effective tool for promoting an important agenda.

Is hacktivism effective?

It’s very difficult for activists in small numbers to bring awareness to the issues that they campaign against. It’s tempting for activists to pull stunts in order to raise awareness of their cause; take the instance of Eddie Gorecki and Jonathan Stanesby, two members of Fathers 4 Justice, who scaled the Royal Courts of Justice dressed as Batman and Robin. Their protest managed to gain national recognition in the press, which rather successfully raised their profile. Days later two-thousand supporters marched in London with a tank!

So perhaps hacktivism is just that – the gimmick that raises the profile of a cause. Anonymous hacktivists have used the Low Orbit Ion Cannon (LOIC), a tool for Distributed Denial of Service (DDoS) attacks, to take down several websites of organisations supporting the Stop Online Piracy Act. The Department of Justice website was taken down as well as those owned by the FBI, MPAA, RIAA. Was it effective? Well SOPA failed, didn’t it?

That said, I wouldn’t attribute the success of the anti-SOPA campaign to Anonymous taking down websites. At most, I could imagine people trying to access those sites being irritated that “heir internet isn’t working properly.” Many of the examples of hacktivism I’ve seen are preaching to the converted; this style of hacktivism does not seem to do much to engage with the public beyond creating momentary annoyances.

I believe that most of the attention to the anti-SOPA campaign came from the blackout of prominent websites such as Wikipedia, Reddit and Craigslist. This seemed a brilliant strategy to confront end-users with the effects of censorship, perhaps more-so than a ‘500 Internal Error’ web page.

However, aside from black-outs, there is also the case of internet vigilantes such as ‘The Jester’ who put a lot of effort into disrupting the websites of alleged terrorist organisations. As of late, he’s also helped to put behind bars several ‘script-kiddies’ who disrupted the UK anti-terrorist hotline as well as disrupt the activities of Anonymous group LulzSec. If there’s such a thing as hacktivism in action – that’s it!

Is hacktivism ethical?

On the face of it, a lot of DDoS attacks can seem to simply be retaliation – an eye for an eye. I’ve never been comfortable with that stance. To me, activism should be about rectification rather than revenge. So can hacktivism ethically meet this criteria?

It could be seen that hacktivism in the form of website take-downs and take-overs is a method of censorship. Denying access to information from groups with opposing viewpoints could be seen as dodgy behaviour, however I’m not ready to brand this as censorship. I feel that censorship is a very strong term describing the suppression of ideas; this sort of website blocking is more comparable to graffiti than book burning. These actions often take websites offline for a few hours, nothing serious enough to cause lasting damage but just enough to raise attention to a cause. Consider as well that this behaviour could be a form of disobedience in refusing to accept services as-is; that perhaps attacks like these are comparable to blockading buildings like the recent protest in Mexico against biased TV reporting or even the Occupy movement.

It’s easy enough to imagine however that the owners of these websites won’t see having their own content effectively blocked as a legitimate act of protest. It’d be all too easy for groups of organised hackers to pick on somebody whose livelihood depends on their web presence. It’s unlikely to be that bad; targets of hacktivist attacks tend to be large multi-national corporations like Visa, to whom a few angry nerds with a DDoS script pose little risk. But the key question as to whether these attacks are justified does not have a blanket answer; it will always depend on the specific case.

A tactic that certainly should be of concern to digital rights campaigners is document droppingor as it’s referred to: “dox drops”. This is the practise of hackers stealing personal, or otherwise private, information pertaining to individuals and publishing it to the web. Personal details of executives of the pro-copyright lobby were published to the world as part of Anonymous’Operation Payback. Clearly that was a breach of individual privacy and served no purpose for pursuing an agenda, other than an incitement of retaliation.

I feel this comes down to an argument of “does the ends justify the means?” If we consider that using hacktivist methods could disrupt terrorist networks or promote a particular cause or ideology, are we willing to accept that it’s ok? At the end of the day, hacktivists have to accept that it’s as important for those they disagree with to share their opinion as it is for those they oppose.

So is hacktivism a genuine form of protest?

I’ve spent a while thinking about this but I believe that yes, hacktivism is a genuine form of protest. Clearly from the attention that has been given to causes utilising hacktivist methods it seems fair to say that hacktivism can be effective. I find it debateable though as to whether hacktivist methods can create sympathy for a cause, it’s difficult to ascertain the helpfulness of hacktivism for a particular cause. I do believe that it is entirely possible for hacktivist methods to be used in a justified and ethical way.

The serious nature of hacktivism necessitates that it be one of the final options of protest for when all other methods have failed. It’s a rather aggressive tactic that’s more likely to intimidate and aggravate rather than promote progressive discourse between two parties. Failing that, clever clogs hackers will no doubt find a way to promote their message.

So anyway, I recently re-installed Ubuntu on my desktop PC. I quite like Linux and I’d want to use it more often but I find that certain things don’t work so well. But anyway, here’s a few of the issues I’ve found with Ubuntu 12.04

Skype

Skype’s come a long way since it bumped up to version 4, the inclusion of a tabbed window interface really helps to streamline everything. Still, sound for 64-bit computers is still an issue. I’m thinking it’s a PulseAudio/Skype issue but all sound is badly distorted. It’s little things like this that compromise the usability for everyday users.

Unity

For all it’s bad press, I quite like the unity interface. The way the top bar becomes the title menus really does a lot to clear up the clutter everyday PC users face. However, I’ve found the integration of other applications to be somewhat lackluster, particularly in the system tray area which blocks other applications from displaying icons there unless you open the console and use gconf to change the white list.

The side-bar ‘dock’ that housed all of the application icons no-longer auto-hides which was a useful space-saving feature. Even better was when there was an option so you could customise it to your own personal preference! I don’t mind Ubuntu nabbing a few Mac OS-X UI features here and there because if there’s one thing Apple are good at it’s UI design (iOS Homescreen and notifications not withstanding); what I do have an issue with is Ubuntu taking Apple’s rather authoritarian approach to software development – there’s no space for personal customisation just like…

Volume Control

Ubuntu tended towards incremental steps of 6% if you change the volume, for headphone users that can be quite significant so if you’re not wanting to be deafened, you’d change that figure. The gconf utility used to allow you to do that just fine and dandy but now that option isn’t there. I got around this by using custom key bindings and this .volumeHack.sh file. I didn’t bother with the xbindkeys script and instead just used Ubuntu’s provided keyboard shortcut manager in system settings.

IM Clients

Empathy is still the default IM client on Ubuntu and I’m alright with that because I feel Pidgin has really lost a lot of ground to Empathy. The one feature Empathy still needs to implement to overtake Pidgin is combining contacts – to this day Trillian is the only IM application that achieves this properly. Combined contacts should:

Be able to be viewed as a mixed chat log from individual contacts, combined chronologically.

Prioritise a particular account protocol.

Easy switching and notifications of changing protocol within a single chat window.

Skype support would be lovely too but somewhat of a pipe dream at this point, though Skypekit is available for Linux embedded devices – surely should work on Linux desktop?

Rhythmbox

I can’t believe this issue made it past testing: the bottom of letters which hang below the line are cut off. Minor I know but mind boggling in it’s simplicity.

Chrome

Right-click on the tab-bar and tell it to use system title bar and borders to avoid displaying two title bars in full-screen mode.

Mounting

There really needs to be an easier way to manage mounted partitions in Ubuntu. Disk Utility is an excellent application but I feel it’s usefulness could be extended to managing mounted partitions, it’d be nice if it would automatically mount a particular partition that I use to store my important files but to do this I have to edit fstab – not too much of an issue for me but it’s minor things like this that will confuse Joe Bloggs.