Category: Free and Open Source Software

I’ve been working with phantomjs recently. It’s a javascript based system wrapper around a webkit backend. This allows one to make requests against a website and run the client side javascript, and make screenshots of the way the webpage would be generated including all css.

I’m using it to generate d3 graphs for testings whereby images of the webpage as saved and compared to expected images.

Of course I’m using python since my websites are all in python and usually django these days. So calling a javascript command and pointing it at a running webserver sounds like a really hard task. But actually django provides a live TestCase which runs a test web server which phantomjs can use as it’s source. It even includes fixtures and other test features which makes it fairly easy to put together the basics of what’s needed.

I haven’t posted in a two years. I don’t know if anyone is still following my blog or not, but if you are. Hello and thanks for your patience.

These last few years I’ve been doing a lot of django contracts. It seems like every man and he’s dog business wants a django website to do complicated things involving user data. And django has matured in the last few years to become more sensible and consistent.

I remember back in 2009 talking with a developer about migrations support and how django had really let us all down by not really having it’s own migration framework built in. Now though, we have what is a pretty good (but not perfect) migration system.

These are some of the things that I have to do for many of the django projects I work on.

django-autotest – A way to run the testing framework over and over again without re-creating the database schema, auto rerun on file changes, shell title bar status updates, some nice base test extras like getting objects, making requests and setting up sessions which often crop up in test code. (include useful haystack base tests too)

django-fast-fixtures – A wrapper for dumpdata and loaddata to have media in fixtures. Useful for those statuses with icons and those tests with media files.

cache middleware – Caching middleware. When using django-cms, it’s VERY important to use the full page cache. Without it, your site is dead (or at least inkscape’s website would be) I’m still working on the expiration logic for all object types.

django-cats – I’ve been trying to convert my haystack/django category and search view into it’s own module. So far, it’s in a collection module I’ve used with multiple projects. The basic idea is that you can make a view that can slice up a queryset into multiple categories and haystack searches. Categories provide counts and easy to display and link iterators for templates. As well as the handy automatic haystack attribute creation and url key word support.

settings and admin tweaks – Most sites need a local settings template. This is because django doesn’t yet seperate out developer configuration from sysadmin configuration. So we still have to do it manually. It also doesn’t allow admin sites to be designed/changed, these two are taken care of in these standard patterns.

urls.py – And lastly is url_tree, a two line addition that ends up in almost all of my urls.py files as it’s just so useful. Take a look at the complexity of this urls.py, it’s made a lot more understandable by reducing the three calls into one simple call. Sometimes I tweak it to allow the view module name in too, but it’s mostly never needed.

I think the core of finding a job and being motivated to do it is: How are you going to serve the people around you? At the end of the day society isn’t about what you can get out of it, but what possible super-power do you have which could effect the people around you in a positive way. Is that art? Is that cleaning? Thinking? Programming? I find that in the modern society too much emphasis is put on person gain, noble voluntarism or doing what’s enjoyable. The problem with these perspectives is that the focus is never on the individuals being served by the work and this blog post is a defence of direct service to others needs and having that be a mark of a successful career.

Doing a Thing

Sometimes something just needs doing. I think if you are in a strong community, you can sense when there’s pent up demand. I ran a campaign to raise funds to get aluminium computer case badges printed and this was before Kickstarter existed. I handled $4k and 20k case badges and they were sent around the world and I had an enormously positive experience serving the people who really wanted Ubuntu case badges. I was transparent about the money. I put the leg work into my relationship with the manufacturer and published news as soon as I had it. I published the designs so everyone knew what they were getting and in the end I had a thousand positive recipients and one negative one because they had been mangled in the post (sorry!) and I used my basic talents using a graphics package to put together something acceptable for users. Anyone could have done this, but to act is to be bold sometimes.

The side effects were more interesting. I asked people for ‘at cost’ payments which despite being based on estimates and the shipping costs wasn’t certain. People were generous and they almost always sent more than asked or needed, often with a note saying “keep the rest”. I didn’t feel bad about this, I think because I was honest with the costs and they were being served genuinely. The other effect was that we introduced the manufacturer to the idea of “open source” by licensing the designs under CC-BY-SA. I encouraged other groups or companies to use the designs to make their own. There are now many, many different aluminium case badges for different Linux distribution and I see them at conferences all the time. I feel good about this. Because imagine all the Fedora, Debian and Mint users who get something cool and the organisers who got a pattern to follow and a manufacturer to talk to. In a way I didn’t just serve people directly with the pre-sales, but also incidentally people through positive open source licensing and I can feel good about both.

Validity of Competence

But then, don’t I just feel good about it because I got praise for it? Well that’s true, I feel good when I get praise. But I think praise is more than just the warm fuzzies. It’s an important reminder to us that our work matters, that we’re still good at it and of the kinds of people who we have served.

One Harvard school gives awards to people who do good work in the anti-alarm professions. You know, those jobs where if the work is done well then no one even notices. But if something goes wrong the alarm bells ring loud. The facilities team, the system admins, the people who order those weird coffee machine cartridges. And that’s a really good thing to do, because it reminds those workers that their job is important and effects probably everyone else in some small way as well as for everyone to thank them once in a while. That feedback from server to served and served to server is the validity of competent altruism. I say it’s altruistic because only thinking about one’s self will not make you great at serving others.

There is a cross-walk near where I live where the snow had been repeatedly ploughed into the pavement one and a half meters high and had frozen solid with thaw. People walking to the train station would have to climb this mountain of snow to cross a very busy road. What if they had kids, or groceries or a bad leg? The night I thought about those people I got the ice breaking tools and shovel and cleared the whole corner. The next day as people shuffled to work, they probably didn’t notice their commute was a little better than before and had no way of knowing if it was the city or not. But I certainly reminded myself of all these slightly less miserable faces as I pounded away at the ice with the sweat pouring off me and the echoes of each thwack, crunch late at night for two hours.

Ethic

I hear about “work ethic” a lot, and it’s usually about “working hard to get on in life”. But anyone successful knows that there’s more to a good work ethic and I believe the largest missing piece is that is “one must work hard for other people, for them and not what one may want to do for them“.

There are communities in the Free and Open Source world who have chosen to work hard at imposing what’s good on their users. That’s a risky service ethic, a sort of pre-emptive competent guess. If your community has set up no way for your guess to fail or you mishandle the push back from your users then you can’t claim to have a good service ethic or to have actually served well. So how can these workers and their work be tied to the people they serve so that it heads in the right direction for the served rather than for the server?

I know that people will pay for good service. there’s no reason to slave away as a volunteer if your work is genuinely in service to others. Caveat, except for people who have nothing, but then grants, government programs, charities can help fill that hole. For my fellow Free Software developers, they do good work and yet many are failing to make the link between the work they do and the people they serve. Thus making it harder for them than it needs to be to earn a living from writing Free Software.

Being Good at what you Like

One part I’ve not talked about is getting a job that you like doing. This is a tricky one because being good at something makes you more likely to enjoy doing it. And Enjoying doing something makes you more likely to do it and thus overcome failures faster and become good at it. I say do what you are good at and like doing that gives the most benefit to others. It’s not as pithy a personal moto though.

A friend of mine who works in mental health once turned a job down for more money. Not just because it would mean serving people who really didn’t need the help, but mainly because being good at helping a certain kind of person with certain kinds of harsh needs is important work. Being good at this job wouldn’t mean there would be lots of recognition from patience, but it would mean truly serving and making a difference to those individuals. And it’s better to have the job one is good at making a difference than the job that could make a difference if only you were good at it or the job that makes no damn difference at all.

And serving individuals I think offers the best perspective on one’s own work. It motivates better than saving the world and feels more accomplished than earning stacks of cash. When one can see the good work well done and be validated for it there is no reason not to go to work every day and earn a living doing it.

I hope you’ve enjoyed hearing from me after a year of no blog post. Please leave your comments below as I love hearing from people.

Today I’ve put the finishing touches on python-crontab 1.4. This release updates the tests and fixes a couple of bugs as well as two really interesting features.

The first is the ability to pull a log from a crontab or cronjob. When creating a crontab in the usual way you can specify a optional log file, if not specified the value ‘/var/log/syslog’ is substituted. The log attribute then provides you a filtered log view which limits the lines returned by the iterator to those attributed to the user your crontab controls. This is of course by default the ‘root’ user, but can be other users too.

Conversely each job in the crontab can return a log iterator which limits the retuned entries to those for this job only. This allows the programmer to tell when cron jobs were last run.

The second large feature is the scheduler. Using the croniter python module, this feature returns date/times when cron jobs would run from any given datetime offset. This allows the programmer to compile a list of datetimes when the job would run in the future and in the past. Use the schedule attribute from the cron job object.

Post was drafted Feb 2nd, delayed for review but is published now without finale edits.

Platforms are everything these days. They drive users in specific, and well structures ways and can make or break different ways of production. Take for instance the World Wide Web, it’s a platform that allows anarchy and it fundamentally breaks the traditional media’s economic model of charging for content per user. The World Wide Web does this by delivering content not just more cheaply, but more quickly and more succinctly than ever before.

By comparison consider iTunes which came well after. A platform which like the World Wide Web is built upon the internet with similar technology. This platform provided a more cohesive and contained experience for getting access to content that users could have gotten through the World Wide Web. Even though it was more expensive to do so, users have bought music and other media through the iTunes platform because the platform is more effective at delivering content to users than the World Wide Web’s anarchy.

The iTunes platform could be credited with helping solve a key economic problem that was befalling the music industry. How to get users to pay for music and thus make music creation worth being in involved in.

We Have Failed Here

Knowing this about iTunes, I’m jealous. That platform has achieved something which we in the Free Software industry have failed to do. That is; meaningfully provide the platform necessary to get users engaged in software production. Economically speaking, we need users who do not make code, but who want to use our software. Socially speaking, if we wish to Free users, we must serve their needs and therefore be willing to be told what they need and deliver on their expectant demand.

This isn’t just about making money for developers so they can quit their proprietary jobs and sustain their lives on making Free Software. This is also about the incredible disservice we give to users. Our ignored users. Those people who we SHOULD be serving with every key press but who we don’t pay attention to unless they morph into helpful bug reporters or fellow programmers. Which plenty do, just so they can be a part of the process.

This is a problem that not only hurts people’s perceptions of Free Software projects, but it also makes our industry weaker than it needs to be. Projects exhibit fragility and an inability to grow. Users pick software on existing features and compatibility and not on future prospects. User involvement is suppressed.

Just Saying No

There are many programmers, project leaders, Free Software members, who ask that money never be involved in Free Software production. These people do not know what damage they are doing to their projects. Not knowing how to get user money into a project is a typical problem, but there is a mindset from the leadership in some projects that having users pay developers should not be allowed. I _do_ understand why money is stigmatised, but this is a symptom of a project’s lack of codified user focus which would provide strong definitions of self-serving, charitable and user bought developer attention. No programmer serving his own needs should be jealous of another getting paid by serving user’s needs. We all need to grow up a little here.

What we need in committed Free Software projects is a meaningful service ethos that makes the users of software the firm target of the project. It would supply direction and impetus to many projects that can’t understand why users don’t like their code and provide economic input to drive projects faster towards those user centric goals.

User Focused not Business Focused

“But Open Source is very economically successful” I hear you say. Well, yes, if your a business it’s been great. The bigger your business, the more meaningful relationship you can have with projects by hiring developers. Having developers (or being a developer) is a sure way to have input. Open Source and the OSI have focused hard on making sure the business to business open source industry works.

I always wondered why Open Source was doing so well and yet doing so badly and it’s this: Big business needs are being met, small business and user needs are not. This isn’t good enough. As a developer if you’re not taking money from users then you aren’t serving user’s needs. Follow the money, follow the demand.

Social Justice

And if your thinking that this economic problem isn’t important for Free Software, think instead of all the users who are disempowered. This is what Free Software social justice is all about. A user is a super important component to development and with continuous development strategies and increasing segregation between developers and non-developers; we need to have thought about getting all users in a position where they can truly be a part of our development practices. They’re the core and source, not the periphery to be ignored.

Demand Change

I’ve been involved with two projects where I see a problem. The first is Inkscape, a project with no economic steam and plenty of users who have no idea how the inkscape sausage is made. It has direction, but no growth. Programmers, but no self-serving power left. Users with needs, but no way to meet them.

The second is XBMC plugins. Here there are thousands of tiny self contained projects and they’re all organised into a forum. Go onto any plugin forum thread and the pattern will be the same: “The plugin stopped working”, “It doesn’t work here either, where’s the developer?”, “I hope the developer comes back to fix it”, “I hope someone else comes in to fix it”, “Has anyone got it working yet”. Again and again, users who are putting their time into begging developers for attention. It’s a depressing situation that must put users off and certainly doesn’t speak well of the stability of XBMC when it’s most useful features are plugins which fail all the time from patchy maintenance.

Demand is important and getting user demand focused in a meaningful way has been our failure. Focusing on support models and business to business open source processes has been our distraction. Harnessing the paradime shifting nature of the platform should be out solution!

A Platform Example

The Ubuntu Software Center is a platform. The way it’s been set up is as a clone of Apple’s iOS store and it’s incompatible with the Free Software industrial process. Instead of helping Free Software it’s driving economic sustainability to proprietary software development and away from Free Software. It’s got unintentional institutional bias which is rooted in the ideas of the developers and managers at Canonical. The suggestion that donations are somehow a meaningful way to drive money to Free Software projects is a sure sign that a person doesn’t ‘get it’, Free Software isn’t a charity case in need of a one off anonymous tip. It’s an industry and with a unique production process that requires careful cultivation and sustainable connections that focus latent user demands to developers and potential developers attention. The USC and it’s makers fail to see that.

Use the User Force

A platform like the Ubuntu Software Center should be made though. But made to specification for the Free Software industry instead of the Proprietary one.

Imagine it embed directly into every Free Software desktop and mobile distribution. The same user focused invitation to join a Free Software project, involving money and time without stigma. Think of a framework available to Fedora, Debian and Ubuntu at the same time, users able to come together and join us in the community by opening their wallets and telling us to get to work on their dreams.

Imagine the power, the vitality and yes, even the vibrancy of the ideas users demand we make for them; all developed into a platform that like iTunes could take a wild west and focus it with good design into a platform that delivers successful sustainability for creators and meaningful dialogue with users.

We need transparency in the accounts of projects to foster trust. Progress of bugs and roadmaps delivered to the desktop so users can see a future in our projects. Reviews and statuses of developers working and available to work. Users electing favourite developers as heros of their causes with monthly payments to kick their bugs before they ever get to the archive. Kickstarter style risk investments to push radical designs and brand new projects. Bug reports where money can be added to the heat to indicate demand for attention and the rewards for completion.

It’s all possible if we dare to make users the center of the Free Software universe and scale it big, VERY big.

We require the courage and vision of the leaders from Canonical, Red Hat, Debian, the FSF and every project leader out there to advocate for User focused Free Software and economic sustainability. With a willingness to embrace our industry’s unique software production method, good design of the frameworks and an invitation to users and paid developers we can make the Free Software industry a powerful and successful part of every user’s computer experience.

Launchpad asked me if I wanted to continue to be an Ubuntu member. I thought about it, and have decided that I don’t. The one thing I’ll miss is being able to post to Planet Ubuntu. But I have to be honest, there isn’t an Ubuntu community any more. There’s a Canonical community, an ubuntu-users gaggle and maybe an enthusiasts posse. But no community that makes decisions, builds a consensus, advocates or educates. It’s dead now, it’s been that way for a while.

Hopefully this post will make it to the planet before my membership expiry stops it. I’ll still be working and using Ubuntu, launchpad, bzr, maybe even ubuntu phone and tv. I won’t stop championing Free Software, economic involvement and good design either. Important principles for me. In fact nothing about what I do, projects I work on will change. This is just a realisation moment that Ubuntu doesn’t have a peer community to be a member of.

Jono,

You were warned plenty. It’s not your fault. You had to deliver decisions against the best interests of the Ubuntu peer community and in favour of the Canonical community. Driving so hard towards product nirvana that peer relationships were driven into the ground. I’m sure you disagree that the community is dead, but eventually those scales will fall or the fake smile will stop. I don’t know what kind of Community you want, but it sure isn’t the peer community I signed up for.

Council,

Unless you can see a way forwards to rebuild this broken dream, disband. Focus your great skills on Debian. I’m sorry we couldn’t make it work, we were overpowered.

Developers,

Keep up the great technical work. If your working on Ubuntu, don’t let the death of the community disrupt you. Let it pass like the breaking of an ulcer and carry on with your important and excellent work. The lack of a community outside of your company is not an impediment or even a problem to making great software. Keep calm and code on.

I’ll be making apps and code, most likely targeting Ubuntu. App developers need not change their behaviour, being an app developer doesn’t make you part of the old peer community. Just devs making good apps that should target all distros. Release your code, don’t get locked in, earn your bread, keep up the good work.

The idea really is that Ubuntu’s rolling release (alpha/betas) shouldn’t become a seperate product from Ubuntu’s LTS industry targets. Instead if we think about the platform as a whole, we should be able to put together both LTS and RR versions of Ubuntu into a single package. It would then be possible to do some really interesting things, like using one ubuntu install to repair or analyse the other.

I’d also like to see most applications removed from universe since it’s pretty obvious that the Debian model for packages is failing and we need to rethink how we deal with applications. The best model would be to be decisive and scrap apps from universe and move them to their own PPAs or a dedicated app-store model where FOSS apps might even be able to earn some money as they would finally be in a similar position to the propritary Canonical apps.

I understand these ideas are bold, but I wanted to give my thoughts on how best to close the loose ends. What are you solutions? colmment below.

This is an example script for python’s jsonrpclib and XBMC Frodo which has an updated rpc library which is finkey about the headers and testy about usernames and passwords. Recorded here to help others in search of a solution.

If you get http errors 415 Media type not found, json decode error, or 401 Not authorised, you need this:

From today’s Q&A with Jono Bacon. I need to use Jono to make a point, so apologies in advance. At 53:00 in the video Jono says is spending 14 hours a day working on Ubuntu. This despite having a brand new baby boy and a awesome meat smoker to contend with.

I say this isn’t fair on Jono. There is no reason why someone should be giving 175%. 14 hours a day is 70 hours a week; That’s more than the 40 hour week enshrined in the European working time directive for office workers; more than the 48 hour maximum safe working week for junior doctors; even more than the 60 hours a week that the Tory government want to change that to.

To put this into perspective, a balanced week would typically or ideally include 56 hours sleep, 40 hours work, 72 hours recreation, responsibility and relationships. But Jono does 70 hours, how can anyone be a good positive cheerleader with that sort of workload?

Responsible Indigestion

I think there is some blockage with the vast array of responsibilities that the Canonical Community Team have to do. A great deal of these responsibilities are likely only held by members of the team because we are unable to find volunteer or non-canonical business people to take them over.

Thinking about the people in the community team (who I love with all my heart), I see a ton of projects, lots of talk and quite a bit of regret at not being able to do everything. And I don’t think it’s right that they should be asked to do everything. Outside of the team, lots of aspects of the community are quiet indeed.

Not the Center

I’ve had some really good chats in the past with Jono about the nature of the community team and how it’s often perceived as being the center of the universe for quite a lot of the Ubuntu community’s activities. There is a misconception that the “Canonical” community team is the authoritative organiser and sanctifier of activities. It’s not.

The Canonical community team are employed to look after Canonical’s interests in the community. So of _course_ they’re going to spend all their time focused on their business objectives, that’s what the team is for. Of course Canonical’s objectives might be over-stretching too, but I think the team also helps out in places for the good of the community which is leading to stress.

Our true centre, if we have one at all, is the Community Council. But it has never really taken on the responsibility of being a co-ordinating power. It deals with problems and that’s what it was made for. So I guess one waits for the project to explode before taking the charred remains to the Community Council for autopsy. This isn’t good enough.

What to do

What we need is a new centre of the Ubuntu Community universe. It’s no good having a company act as the sol centre of activity; volunteers hate working for free for mega-corp big bucks and move away over time as co-ordination is often innocuously prescribed along business objectives. A community needs a strong central point, or central set of core ideas that we can all dance to without fear we’re being abused by a large player using cheat codes.

“Ubuntu for everyone” used to be enough. A rallying cry from a thousand frustrated Linux users to come together, creating a gravity1 all of their own, pulling more people in until we had such a strong and healthy community.

That isn’t enough any more. We’ve been too successful. We have an awesome Linux desktop which we can install on anyone’s computer without very many problems. You can get it for free if you ignore the skull, download an iso good for a million installs of the same fully working operating system.

What we need is a new centre of the Ubuntu Community universe. A new point of gravity. A new co-ordination body who can sanction every hair brained scheme2, listen endlessly to every kid on irc who has an exciting new idea, comment on things, and act as a great mixing bowl for people and ideas. Such a body would need a reason and a goal, something resonant that goes beyond just technical aims, or marketing PR.

I seriously propose that we found an identity for Ubuntu which lies outside of Canonical. A gravitational body of such stability and neutral authority that the massive Jovian mass of Canonical will happily find it’s lagrange point while smaller congregations will find more comfortable points in their own orbits.

I hate to say it, but Catch 22, Canonical MUST be committed to such an idea. There is NO point in a bunch of community folk wandering off into the brush to build this. We would need everyone to recognise the problem we have here and help fix it with some radical rethinking about the community. And that includes helping Jono3.

Are you onboard? What do you think? Let me know below in the comments.

1 Gravity, the reason we all come together. Each Canonical PR blunder acting like stellar wind pushing more people away.2 Every hair-brained scheme in Ubuntu is already sanctioned, but most of the time someone with a fancy hat and an official looking title just has to listen to it and say that it’s a good idea to give the kids confidence.3 Save the Cheerleader, save the world.