I agree with that statement, but I found the following source (from evolutionary biology) framed the issues around Brexit in a new way for me. I'm going to simplify my argument somewhat by assuming two groups and characterising them as follows:

Brexiteers: Mainly white british, rural or Northern, down on their economic luck, anti-immigration.

Remainers: Cosmopolitan, urban or from London, not necessarily rich but doing ok, comfortable with the idea of immigration.

I'm ignoring the public school free market liberals who actively want to return to some imagined glory days of swashbuckling empire for my purposes.

With that in mind I'd like to present the following two sources with a huge pinch of salt

Monkeys

RECIPROCAL ALTRUISM

Kin altruism exists because it promotes the survival of one's relatives; but not all altruistic acts help relatives. Monkeys spend a lot of their time grooming each other, removing parasites from those awkward places a monkey cannot itself reach. Monkeys grooming each other are not always related. Here reciprocal altruism offers an explanation: you scratch
my back and I'll scratch yours.
...

Imagine that a species is divided into several isolated groups -- perhaps they are monkeys whose terrain is divided by rivers which, except in rare droughts, are too swift to cross. Now suppose that reciprocal altruism somehow appears
from time to time in each of these groups. Let us say
that one monkey grooms another monkey, searching for disease-carrying parasites; when it has finished it presents its own back to be groomed. If the genes that make this behavior probable are rare mutations, in most cases the altruistic monkey would find its kindness unrewarded; the groomed monkey would simply move away. Grooming strangers would therefore bring no advantage, and since it leads the monkey to spend its time helping strangers instead of looking after itself, in time this behavior would be eliminated. This elimination may not be good for the group as a whole, but as we have seen, within the group it is individual rather than group selection that dominates.
Now suppose that in one of these isolated groups it just
happens that a lot of monkeys have genes leading them to initiate grooming exchanges. (In a small, closely related group, kin altruism might bring this about.) Then, as we have seen, those who reciprocate could be better off than those who do not. They will groom and be groomed, remaining healthy
while other members of the group succumb to the parasites.
Thus in this particular isolated group, possessing the genes for reciprocal grooming will be a distinct advantage. In time, all the group would have them.
There is one final step. The reciprocal grooming group now has an advantage, as a group, over other groups who do not have any way of ridding themselves of parasites. If the parasites get really bad, the other groups may become extinct, and one dry summer the pressure of population growth in the reciprocal grooming group will push some of its members across the rivers into the territories formerly occupied by the other groups. In this way group selection could have a limited role-limited because the required conditions would not often occur-in the spread of reciprocal altruism.
If we are prepared to allow group selection a role in the inception of reciprocal altruism, we can hardly deny that the survival of some groups rather than others can provide an evolutionary explanation for a more general tendency for altruistic behavior toward other members of a group. This is still quite distinct from the popular view of traits evolving because they help the species survive-groups are far smaller units than species, and come in and out of existence much more frequently, so group selection is more likely to be an effective counterweight to individual selection than is species selection. Nevertheless, a group would have to keep itself distinct from other groups for group altruism to work -- otherwise more egoistically inclined outsiders would work their
way into the group, taking advantage of the altruism of members of the group without offering anything in return. They would then outbreed the more altruistic members of the
group and so begin to outnumber them, until the group would cease to be more altruistic than any other group of the same species. Although this would cost it its evolutionary advantage over other groups, there would be no mechanism for stopping this. If the group altruism had been essential to the group's survival, the group would simply die out.
This suggests that group altruism would work best when coupled with a degree of hostility to outsiders, which would protect the altruism within the group from penetration and subversion from outside. Hostility to outsiders is, in fact, a very common phenomenon in social animals. Although there is a popular myth that human beings are the only animals who kill members of their own species, other species can be as unpleasant toward foreigners as we are. Many social animals, from ants through chickens to rats, will attack and often kill outsiders placed in their midst. In a series of experiments conducted on rhesus monkeys, it has been shown that introducing a strange rhesus monkey into an established group aroused much more aggression than either crowding
the monkeys or reducing their food supply. Admittedly, keeping strangers away could just be a means of protecting one's own food supply and that of one's kin; but it could also be that this behavior serves the same role as geographical isolation in protecting the altruism of the group from debasement.

I realise this is a dangerous extract in isolation - the book goes on to argue that we, as humans, are better than monkeys and that we can and should work toward expanding our inner circle so that we can all include each other. However, it did strike me that it might help to explain the mood of the Brexiteers in some small way. A perceived "threat" to our "food source" would obviously increase hostility, if your narrative suggests that you have to share your limited resources with outsiders. The economic downturn, the language of austerity (particularly from George Osborne), the D***y M**l etc. all fed into that mood.

The Metropolis

Athens was not just a city, in our sense of the term, but a polis, or city-state. All of the poleis of ancient Greece — of which there were, according to the best modern tally, 1,035, though not all were coexistent — were independent states, with their own (very active) armies and their own forms of government.
Each polis had a city center, its astu, usually walled and containing an acropolis, or “city on an extremity,” high up and thus defensible, probably the reason that the original settlement had grown around it. The astu was surrounded by extensive territories, the khora, which included farmlands, olive groves, vineyards. The more established cities — Athens, Sparta, Thebes, Corinth, Argos — also had colonial settlements, other cities that they seized and that would pay taxes to the "metropolis", the mother polis, and were required to be allies in war. Most poleis weren't large. The average number of citizens was between 133 and 800. Athens had a citizen population of about 30,000 while its total number of inhabitants was in the vicinity of 100,000, which means that only one out of three residents held the rights of citizenship. Though there were no property qualifications, as there were in Greek oligarchies, citizenship in the Athenian democracy was hard to come by. Women, children and slaves were excluded from citizenship, as in all poleis. So too were foreign residents, the metics, who were often among the richest of those living in Athens.
Athenians prided themselves on the myth that they, alone of all Hellenes, were autochthonous, literally "sprung from the earth", by which they meant that they had always occupied the same soil. Being born of an Athenian father had long been a requirement for citizenship, but in 451 BCE Pericles tightened this law, the pride in autochthony having strengthened following the the Persian Wars and Athenian imperial hegemony. Now citizenship required both father and mother to be Athenian-born, making citizenship even more exclusive and desirable, just as Athens was asserting itself throughout Hellas as the standard for what made all Hellenes great.

This extract made me think about London as the modern day Athens. For the majority of Athenians, they would have lived in smaller rural communities presumably with clearly defined social groups. Athens (the metropolis) was an order of magnitude larger with around two thirds of its inhabitants being full citizens. There are some further calculations around the fact that women and children are included in that two thirds, but it's reasonable to assume that Athenians were familiar with seeing "outsiders" (metics) in their day-to-day lives. It strikes me that this must have been an economic arrangement that worked for everybody, resulting in a feeling of optimism and reduced hostility towards those foreigners.

Anyway, I don't really have a point. I just thought those passages were interesting.

]]>(Warning: whilst I work for Heroku, this isn't official supported - it's just something I discovered in my spare time. Hopefully this helps someone but leave me any feedback on Twitter @xavriley)

What do you do if you're deploying an app on Heroku but you have a package your app

]]>https://www.xavierriley.co.uk/compiling-packages-from-source-on-heroku-using-buildpacks/4ee4fc39-53f0-47cc-8616-7ac7387834fcWed, 10 May 2017 08:25:08 GMT(Warning: whilst I work for Heroku, this isn't official supported - it's just something I discovered in my spare time. Hopefully this helps someone but leave me any feedback on Twitter @xavriley)

What do you do if you're deploying an app on Heroku but you have a package your app depends on? The first thing is to check the list of default packages here https://devcenter.heroku.com/articles/stack-packages - if it already exists there then you don't need to worry.

The next thing to checkout is the list of third party buildpacks here https://elements.heroku.com/buildpacks These are contributed by the community and it's fairly likely that someone has gone down this path before.

If there's no third party buildpack, or if the ones you've found are no longer maintained, then the next option is the Apt buildpack https://elements.heroku.com/buildpacks/heroku/heroku-buildpack-apt This allows you to create a file named Aptfile at the root of your project which lists packages that you'd like to install. Bear in mind that this doesn't resolve dependencies in quite the same way as apt-get install so you'll need to list all the dependencies you want.

Right - with those out of the way, what about if none of them work? I came across this recently where someone wanted to use a version of curl that had support for HTTP/2 requests.

Finding libraries in the right places

The missing piece of the puzzle was to use the --disable-shared flag with ./config. When the app is building on Heroku it uses funky folders in /tmp and the compiled binary tries to link against those paths. By disabling shared libs, it pulls in all the necessary code to the resulting binary so you can move it around the system without problems.

You will then need to add these new files and deploy. This should build a version of curl for you in the root of the project.

The curl binary and object files are located in /app/curl-7.46.0/src/ (Heroku deploys your code to the /app folder by default).

Note that the dyno now has two versions of curl installed! If you need to use the one with HTTP/2 support you have to be careful and specify the full path ./curl-7.46.0/src/curl --http2 -I https://nghttp2.org/ or add it to the front of the $PATH variable.

]]>(Warning: whilst I work for Heroku, this isn't official supported - it's just something I discovered in my spare time. Hopefully this helps someone but leave me any feedback on Twitter @xavriley)

Update: this won't work

After some more research, the advice below will allow you to install the relevant

]]>https://www.xavierriley.co.uk/using-openvpn-from-a-heroku-dyno/83f675c8-4a90-4dd2-a819-715c27a13c29Tue, 09 May 2017 15:04:06 GMT(Warning: whilst I work for Heroku, this isn't official supported - it's just something I discovered in my spare time. Hopefully this helps someone but leave me any feedback on Twitter @xavriley)

Update: this won't work

After some more research, the advice below will allow you to install the relevant packages but the Heroku dynos won't have the necessary permissions to rewrite the outgoing network packets. For an alternative way of setting up a secure tunnel to a dyno, you could try looking at https://github.com/koding/tunnel or https://github.com/jpillora/chisel instead.

]]>In a 1993 paper by Anders Ericsson, he was one of the first to make the claim that mastery of a skill took around 10 years (about 10,000 hours) of deliberate practice.

In most domains of expertise, individuals begin in their childhood a regimen of
effortful activities (deliberate practice)

]]>https://www.xavierriley.co.uk/10-000-hours-as-told-by-jazz-musicians/27ceaee6-6998-4580-a73f-e7dedec897deThu, 13 Apr 2017 12:17:28 GMTIn a 1993 paper by Anders Ericsson, he was one of the first to make the claim that mastery of a skill took around 10 years (about 10,000 hours) of deliberate practice.

In most domains of expertise, individuals begin in their childhood a regimen of
effortful activities (deliberate practice) designed to optimize improvement. Individual differences,
even among elite performers, are closely related to assessed amounts of deliberate practice. Many
characteristics once believed to reflect innate talent are actually the result of intense practice
extended for a minimum of 10 years. http://projects.ict.usc.edu/itw/gel/EricssonDeliberatePracticePR93.PDF

Whilst most people reading this will probably be fellow computer programmers clicking around from my other posts, I'm also a jazz guitar nut and have been for some time. I thought it would be fun to look at quotes from jazz players to see what they can tell us about getting better at something.

Whilst lots of people subscribe to the "talent" theory of development, that dream for me died a long time ago. Instead I feel like the stories of excellence (particularly in jazz) almost always have these common themes:

starting young

taking lessons with a good teacher early in their development

some period of intense practice, especially for those with great chops (technique)

lots and lots and lots of gigs

Let's see some quotes to back that up!

Joe Pass

Was guitar playing easy for you in the beginning?

I guess it came sort of easy for me; I have certain difficulties, not a lot. But you've got to remember that I grew up playing the guitar. I started when I was nine, and by the time I was nine and a half or ten, I was doing seven or eight hours' practice every day. I did two hours' practice at six o'clock in the morning before I went to school, and another two hours as soon as I got home from school in the afternoon. Then I did four hours at night before I went to bed.

I did that until I was fourteen or fifteen. I didn't like it - I hated it, but my father was very firm about it; he saw a little something happening, so he figured he'd just push. I don't remember too much how I felt about it except that I'd rather be outside playing ball and things. I never could ride a bike, like even today I can't do these things.
Joe Pass Interview Guitar Magazine in June 1974

8 hours a day for five years, all before the age of 15!

But one wouldn't recognise any resemblance between your playing and Django's,

Well I never copied him. I don't remember that I copied any guitar player note-for-note. But I remember copying Charlie Parker note for note.

George Benson

I'm writing this in 2014, a year that, professionally speaking, was pretty much the same as 2004, which was pretty much the same as 1994, and so on all the way back till the 1950s. And I don't mean that in a bad way - it's more about the magic of consistency. See, I'm a creature of (jazz) habit: I practice every day, I gig a lot, and I record almost every year. I keep going.

I also remember reading a quote in a YouTube comment, possibly on a Jack McDuff video, (so sad that folk histories are locked up in there, mixed in with vile trolling etc. completely unsearchable) that someone had a friend who knew Benson in the early days. They claimed that he could sing around 20 (or 21 or 22, can't remember exactly) Charlie Parker solos from memory when he was around 18.

Charlie Parker

I put quite a bit of study into the horn (saxophone), that's true. In fact, the neighbours threatened to ask my mother to move once... She said I was driving them crazy with the horn. I used to put in at least 11 to 15 hours a day.

He was around 15 at the time as far as I can tell. The same book describes him learning Lester Young solos note-for-note.

Pat Metheny

and of course, you HAVE to practice like crazy. during the years between when i was 13 til 19, i would guess i averaged around 10 to 12 hours a day with the guitar in my hands or sitting at the piano studying harmony.

]]>Now that Ruby has crested the hype cycle, settled down and taken out a mortgage, you'd expect the posts around the community to be more about big business concerns. Whilst that might be true, I'd like to row against the tide by telling you about the fun I had figuring]]>https://www.xavierriley.co.uk/writing-a-c-extension-for-ruby-in-2016/e621d3e1-c057-4ecf-9fd0-52de11315dd4Sat, 12 Nov 2016 22:28:07 GMTNow that Ruby has crested the hype cycle, settled down and taken out a mortgage, you'd expect the posts around the community to be more about big business concerns. Whilst that might be true, I'd like to row against the tide by telling you about the fun I had figuring out how to write my first C extension - fast_osc - and how I made Sonic Pi 10x faster in the process.

Before we get stuck into the details of that, what is a C extension and why would you want to write one in 2016?

Why?

Two reasons - speed and leverage. If you have a need for speed and a small well defined problem then writing an extension may be the way to go. Whilst working on Sonic Pi we have to target the resource constrained Raspberry Pi Model B+ . With a 700MHz CPU and 512Mb we often have to party like its 1999. After some recent profiling work we identified that the Open Sound Control message encoding/decoding was a big hotspot in terms of performance and this was causing audio dropouts as the CPU struggled to keep up with our example pieces.

Speed

We optimised as far as we could with plain Ruby, but our implementations relied on Array#pack and String#unpack to get data in and out of the right binary format. If you take a peek at the View source links for these at the bottom of those pages, you might be able to see why they were a bit slow. The implementations run to hundreds of lines, many of which we didn't need as OSC only uses a limited number of types.

Leverage

Open Sound Control is a widely used messaging format in music applications so there are plenty of good implementations in most languages. One of the great use cases for a C extension is when you don't want to reinvent the wheel as you can use the great work of others instead.

Before diving in...

Writing a C extension is a little bit like doing distributed computing - it's best to avoid it until you have no other option. I'd initially wanted to use FFI (which I'll talk more about in another post) because, frankly, I was scared of writing any C that was going into a widely used app.

The issue with FFI is that it largely depends on the library you want to wrap - if it has C macros and structs flying around all over the place then it's likely to be more work writing the FFI wrapper than just writing the C extension in the first place.

It's also worth mentioning that there are great options for writing Ruby extensions in Go, Rust, Crystal and many other languages that can expose C bindings. I didn't choose to go with those for this project because the OSC libs weren't as mature but I'd definitely consider them in future.

rtosc - nice, clean C

After looking at several OSC implementations I was starting to despair - I'd never written any C before and the canonical implementation, liblo, made use of macros and structs that I didn't completely understand at the time. After nearly giving up I stumbled on the rtosc library which had embedding in mind. It consisted of two files (rtosc.h and rtosc.c) and emphasised speed by pre-allocating a buffer for the message so that assignments were kept to a minimum.

In the best spirit of open source, I enquired with the author via a GitHub issue and got a thoughtful, well-reasoned and polite response within a few hours - viva la internet!

This managed to convince me to try writing a C extension and I'm glad I did.

Open Sound Control and the fast_osc API

OSC is a binary protocol which looks a bit like RPC. There's a "path" which represents the name of a remote method, a tag string which identifies the type of the args and then a series of binary encoded arguments. You can think of a basic message looking something like this:

/methodname, "sif", ["This is a string", 123, 3.14]

You can see that the type of the arguments corresponds to a tag - s for a string, i for an int and so on. For the Ruby API I wanted to hide this so that you weren't thinking about types e.g.

This meant that the C extension would have to take a Ruby array, do some kind of switch case statement on the type of each element, and finally build a tag string and an output array of the encoded versions according to the OSC spec.

This is pretty much the whole challenge but it did mean I need to touch Ruby's mysterious C API for quite a few different types. You can checkout the source to see how these fit together but I'll comment on a few here:

This little snippet made sure that the decoded string was passed back to Ruby as UTF-8 rather than US-ASCII - phew!

Ruby Time objects

OSC also has a concept of timestamps which are pretty integral to music applications. I wanted the API to transparently convert these to and from Time objects. First you have to check for the object type:

case T_DATA:

This means that you have a Ruby object. From there we check specifically for time using this:

if (CLASS_OF(current_arg) == rb_cTime)

Now rb_cTime is one of the many examples of why I called the Ruby C API mysterious. This was quite tricky to find out and one of the main reasons I'm writing up this log to aid future intrepid explorers.

OSC Timestamps (NTP)

Finally, the OSC spec encodes time in two parts - first as the number of seconds since the year 1900 with the other part for the fraction of a second passed. This is what NTP servers use and it differs from the more common "seconds since 1970" (Unix epoch) that Time.new and others deal with.

The method for this was a bit hairy but after working out that I needed to Google "Unix to NTP timestamp conversion" I found something close enough on Stack Overflow and adapted it to Ruby like so:

Memory management

Thankfully this wasn't to complicated as the underlying rtosc library just expects you to allocate a buffer big enough to hold the message you want to encode. This meant keeping track of the size for each of the different types above and summing them for a given message. For strings (which vary in size) I took the bytesize() and then rounded up to the next power of two just to be safe.

Conclusion and further reading

After a lot of Googling, head scratching and running rake compile I finally had something that worked but I had no idea whether it was actually any faster. Thankfully, the benchmarks came in and they were good:

oscruby here is the non-optimised library for Ruby. samosc is our hand rolled optimised pure Ruby version and fast_osc is this C extension. As you can see there was something like a 4x improvement in Encoding to OSC and a 10x improvement in Decoding over the fastest pure Ruby implementation.

This new extension shipped with Sonic Pi v2.11 last week and has been battle tested in 1000s of live coding performances without missing a beat, so I can say that it was a complete success! (it's not often I get to say that about software projects...)

One that does deserve a special mention though is the The Ruby Cross Reference maintained by whitequark which offers a search over all the symbols in the Ruby C source. I found this helpful for some of the more obscure parts (like rb_cTime) but sadly this resource appears to be shutting down on 2016-12-31 unless it finds a new owner - any takers? If you're interested you can contact them via the banner at the top of the site.

If you've been inspired to write your own extension, or if you have any feedback on my awful C I'd love to hear from you - I'm @xavriley on Twitter - thanks for reading.

You can try this experimental Heroku Apt buildpack. The documentation is a little out of date as it's best to use it with the multiple buildpacks support. Buildpacks are order dependent so you'll want to add this one first in the list:

Why not just use libboost-all-dev?

I'm so glad you asked. The Heroku Apt buildpack isn't the same as apt-get or aptitude so it doesn't have any dependency resolution. That means you have to specify every dependency in the Aptfile.

]]>Working with Sonic Pi one of the exciting things about being able to code music is the potential for using algorithms. For the non-technical reader, I like to think of algorithms like following a recipe. You might start with the same ingredients but the order in which you do things]]>https://www.xavierriley.co.uk/neutron-accelerators-and-drum-machines-with-sonic-pi/040314de-035a-49f3-bdc9-6b13cc0c4876Thu, 22 Jan 2015 12:00:00 GMTWorking with Sonic Pi one of the exciting things about being able to code music is the potential for using algorithms. For the non-technical reader, I like to think of algorithms like following a recipe. You might start with the same ingredients but the order in which you do things can affect to outcome.

When we think about music people often understand things at an intuitive level but become uneasy about the idea of using rules or recipes to generate it. "Formulaic" is a dirty word when used to describe pop songs for example, but it doesn't have to be that way. I'm going to look at one particular algorithm, Euclid's algorithm, to create musical rhythms.

A method for spacing out rhythms

Despite all the complex words I'm about to use, if all boils down to this: if I have a number of beats that I want to play in a given space (say 2 bars), how do I space those out as evenly as possible?

That's obviously easy with regular rhythms that divide cleanly. If I want to play 8 beats across two bars of 8th notes it would look like this:

x . x . x . x . | x . x . x . x .

That's 8 x's spaced across the 16 possible beats. Easy peasy.

But what about numbers that don't divide cleanly, for example, 3 into 8?

Let's start with the solution first:

x . . x . . x .

You can see we have to use a combination of long and short notes to fill the bar evenly. The above pattern uses groups of 3's and 2's to fill the space. You could easily notate it as:

332

if you count the distances from one beat to the next. Let's look at a few more examples of these odd rhythms:

The last of those options appears so often in so many kinds of music that it's often called "the mother rhythm". Everything from Cuban clave to Elvis' "Hound dog". That said, if you look closely they are all variations on the same grouping: 2 lots of 3 and one lot of 2. That means we only need to generate one version and then we can rotate it to produce the others. So how are we going to generate it then?

Back to the future: Euclid's algorithm from 300 B.C.

It turns out that Euclid thought about this too - how do you space things out as evenly as possible? Thousands of years later a scientist named Bjorklund was facing the same problem when he was setting up a Neutron Accelerator. He needed to fire neutrons evenly in a given space of time (a bit like our rhythm problem) and, being a scientist, he came up with a clever way of doing that.

That's the algorithm represented in Ruby. You don't have to understand it as it's probably not the simplest way of explaining it, but it's there to show how concise a problem like this can be when expressed in code.

From here it's not too hard to map over those arrays to turn them into musical patterns. Here's an example you can play with:

comment do
# this function is available in SonicPi v2.4 and upwards
# if you're using a version with this included you can
# delete this distribute function, otherwise uncomment it
# above
def distribute(accents, total_beats, beat_rotations=0)
res = []
total_beats.times do |i|
# makes a boolean based on the index
# true is an accent, false is a rest
res << ((i * accents % total_beats) < accents)
end
res.ring
end
end
# Monkey patching is never a good idea
# just say no kids...
class SonicPi::Core::RingArray
def as_x_notation
self.to_a.map {|x| x ? 'x' : '.'}.join(' ')
end
def as_beat_groups
self.to_a.slice_before {|x| x }.to_a.map(&:count).join
end
end
use_bpm(120)
def play_sample_for_sequence(pattern, sample_name, sleep_time = 0.25)
pattern.each.with_index do |beat, i|
sample sample_name if !!beat
sleep sleep_time
end
end
uncomment do
live_loop(:hh) do
with_fx :level, amp: 1 do
cue :heartbeat
play_sample_for_sequence(distribute(11, 16), :drum_cymbal_closed)
end
end
live_loop(:bd) do
sync :heartbeat
play_sample_for_sequence(distribute(5, 16), :drum_bass_hard)
end
live_loop(:sn) do
with_fx :level, amp: 1 do
cue :heartbeat
play_sample_for_sequence(distribute(2, 16).to_a.rotate(4), :drum_snare_hard)
end
end
live_loop(:bass) do
with_fx :level, amp: 1 do
use_synth :tb303
cue :heartbeat
distribute(3, 8).each do |beat|
play scale(:a2, :minor_pentatonic).choose, release: 0.3 if beat
sleep 0.5
end
end
end
end

Here's a demo of me live coding with the above:

Why does this work?

Musically I think it's fair to say that last example sounds pretty cool. But why? I think that the appeal of these rhythms ("Euclidean Rhythms" as Toussaint calls them) boils down to this: because the spacing is always even, when we layer these kinds of beats on top of each other we get strong "cross rhythms". In the example above I've deliberately put the snare on beats 2 and 4 to provide a strong backbeat. All the other rhythms bounce off the strength of the snare rhythm.

I think another possible reason is that our brains enjoy patterns and order even if we can't quite tell what the pattern is. Imagine the difference between two knitted jumpers, one with a completely random colour for every stitch, the other with a strong geometric pattern. Psychologically we're more likely to prefer the geometric pattern (unless the random knit happens to be really cool).

Having these functions in our toolkit means that we can reach musical sounding results even faster, without having to rely on things being totally random but still retaining an element of surprise.

What do you think? Does it sound cool to you? Will algorithmic music ever be widely accepted? Hit me up on twitter @xavriley if you have any comments or questions.

Further rhythming

This topic has already been covered in several other places, but not as far as I know in Sonic Pi yet. There's a very cool HTML5/Javascript version of this kind of drum machine available at http://www.groovemechanics.com/euclid/ which has been on Hacker news. Also there's a patch for Max called Polyrhythmus which is along the same lines. I wanted to cover the musical aspects behind the algorithm in a bit more depth for this post. There's also a good Wikipedia page on Euclidean rhythm with links to other resources.

]]>There are many solutions for getting data from pdfs. I'm going to describe how to use the excellent Java library PDFTextStream by Chas Emerick (of Clojure fame) to get data out of tricky pdfs.

Why PDFTextStream?

Quite simply, it's the best PDF extraction library I've come across in terms of

]]>https://www.xavierriley.co.uk/getting-data-from-pdfs-with-jruby/d3688937-2f3c-492b-8e36-4b473ff4851fMon, 06 Oct 2014 11:00:00 GMTThere are many solutions for getting data from pdfs. I'm going to describe how to use the excellent Java library PDFTextStream by Chas Emerick (of Clojure fame) to get data out of tricky pdfs.

Why PDFTextStream?

Quite simply, it's the best PDF extraction library I've come across in terms of features and performance. It handles layouts and formatting very well and the xml output gives some useful tags for data extraction.

Getting the library

Head over to http://snowtide.com/downloads and download the latest Java version (2.7.0 at the time of writing) and unzip into a folder called jruby-demo.

Some JRuby/Java interop

Create a file at jruby-demo/pdf-extractor.rb with the following contents:

Extracting some text

and after a few seconds of jvm warm up time you should start to see text on STDOUT.

Different extraction modes

standard - This handles column layouts (common in pdfs) and reflows them to make sure the text reads in the correct order.

visual - This preserves the text spacing on the page which is useful for tabular data.

xml - If the source data has bold or italic text, this processor outputs xml markup which can be useful for further processing with Nokogiri or other similar libraries.

]]>Recently I caught a conference talk by @rbin of SendGrid where he outlined some benefits of the "Open Source" movement for the uninitiated. Whilst it was a good talk, something about it lodged in my mind. For me the virtues of Open Source aren't as clear as the speaker made]]>https://www.xavierriley.co.uk/the-evils-of-open-source/6339cb57-129f-4247-902b-f56d17fbfa76Mon, 06 Oct 2014 11:00:00 GMTRecently I caught a conference talk by @rbin of SendGrid where he outlined some benefits of the "Open Source" movement for the uninitiated. Whilst it was a good talk, something about it lodged in my mind. For me the virtues of Open Source aren't as clear as the speaker made out and I'd like to articulate why, based on some quotes and tweets:

Exhibit A: Steve Klabnik on Open Source

Open source is a mega-capitalist conspiracy to reduce programmer wages. ... it's yet another method to drive down costs. ...

The other day I was hanging out with two people and there was somebody there using a jackhammer outside where we were. I said, "Nobody asked the jackhammer dude to work on an open jackhammer project on Saturdays, that would be totally absurd." But that's what we do. Programmers as a class systematically devalue their own labor, doing it for free without being asked to.

It's not universally bad - it's really cool that you can get all this software for free. But, value is being generated from your labor and it's not being captured by you, and that's unsustainable. I'm only able to do open source work now because I'm getting a real salary from Mozilla. They're a one-of-a-kind organization that can afford to pay people for open source stuff ... but the whole world can't work for Mozilla.

He goes on to say how an open source portfolio is now a key job selection criteria which is setting the balance in favour of those who have the free time to work on those projects.

wat?

Here we have one of the most prolific Open Source contributors of all time calling out Open Source as a (mostly) bad idea. Particularly when viewed from a marxist standpoint. This begs a questions about when and where "Open Source" steps over the line into being bad, which I'll try to answer.

Exhibit B: Heather Gold - "Designing for conversation"

Nothing brings people together like a shared problem. This is what holds people together. Part of the reason we solve problems isn't just to build a bridge to get across the river - it's because the feeling of doing it together is awesome. That's probably one of the best feelings we have in life.

If we can agree on this premise, I think it's the best explanation of why we as programmers choose to give up our labour for free. Collaborating on a shared problem is awesome. It feels great. Github is built on the fact that having a pull request accepted, or code you've written being used by others, is a universally good feeling. But this feeling is open to abuse when other forces get involved.

Evil #1 - the "awesome feeling" gets co-opted by evil actors

Here I'm mainly talking about corporate hack days. A while ago, I saw a tweet about Unilever running a weekend hackathon. This upset some people and ISTR it making it onto Hacker News.

The problem here is that Unilever have a market cap of around $74 Billion dollars. They could afford to pay every single attendee of that hackathon a handsome wage for their time and ideas and yet they don't.

As programmers, we need to become aware of these issues like Klabnik is telling us to. Hackathons are not evil by nature - at least with doing a hackathon for charity or the government where you can cite some feeling of civic or moral duty. Even at a stretch, a corporate hackathon might be worthwhile if it presents some valuable opportunity for learning. I can fully see that if I was interested in learning how email tech works at scale, attending a hackathon hosted by SendGrid would be a good use of my time. The same can't be said for Unilever in my opinion. If you're attending a hackathon or contributing to an open source project ask yourself - "do these people value my input with the proper respect?"

Evil #2 - we programmers lose sight of the "awesome feeling"

A topical example. Nick Sutterer is the developer of the popular Active Form plugin for Rails. He recently tweeted this:

What appears to have happened is that some other developer has decided Active Form is better than the current defaults in Rails and copied the project wholesale without attribution. The incentive to be a "core rails" committer is a strong one and I can see that, for those without a moral compass, stealing code from another established project is a tempting option. The result? Another dev has "sold out" @apotonick in order to improve their rep in the Open Source world.

What's interesting to me is that some of the comments appear to be questioning this action on the technicalities of the licence involved. That might be appropriate in a commercial transaction, but in an act of social collaboration? I'm not sure. Everyone is giving their time for free and that needs to be respected - the other dev has denied @apotonick the feeling of pride and satisfaction that would have come from seeing his project merged into Rails. We should all learn from this example - the "Open Source spirit" is built on a bedrock of attributing people properly for their hard work. If we don't call this kind of behaviour out then the whole thing will crumble.

Evil #3 - Open Source != sustainable

As a rule, working for free isn't sustainable. That said, in an ideal world we all have a bit of leisure time to spend as we please. If I feel like contributing to an awesome Open Source music programming project and I have the time, that's my perogative. I think this third evil arises when the motivations for contributing get mixed up. When we're not clear about the boundary between hobby projects and work projects things start to come unstuck. If I start out writing a new super fast web framework as a hobby project (perhaps to learn more or because I thought I could do better than what's out there) that's fine. But let's say that project gets picked up and used by thousands of businesses around the world. This project has made the leap from bedroom hacking to critical infrastructure - success? It is a success of sorts. The good feeling of having your contribution used in the world is undeniable. However, Open Source doesn't have a notion of responsibility here - it's quite likely that people will hold you personally responsible for your projects bugs and failures.

Here's a real world example of TJ Holowaychuk who was so prolific that this Quora post questions whether he really exists. He wrote a massive amount of widely used node js infrastructure and then left it to the community to maintain. To reiterate, that's his right to spend his time as he pleases and there's nothing wrong with that. Perhaps he needed to take a paying job? Whatever the reason, he's one of a string of dropouts from the Open Source community which suggests that burnout is a real hazard both to the individual and to the businesses that rely on their work.

What's the solution here? Speaking with some tech people at Wikipedia they seemed to be intrinsically aware of this. Their tools for wikimedia users and developers are built on the premise that if it becomes valuable infrastructure then the Wikimedia foundation will shoulder the responsibility if you choose to abandon the project. For this reason they require that you host on their infrastructure (for free) and publish the source of your work. This helps to take some of the heat off.

Another solution might be GitTip and other ways of recognizing contributions financially. Whilst these are good, I suspect that they lead to a real world undervaluation of a person's contribution.

The last point is that for Open Source to be sustainable, it should probably be sponsored or supported by an organization. The idea of Wordpress contributing back to ElasticSearch through open source is an entirely natural one. Joint ventures and partnerships are common in the business world. Steve Klabnik is supported by Mozilla and appears to be comfortable with this arrangment. I think that as developers, we should develop a sense of which projects are appropriate for hacking on personally and which are better suited to some kind of organisational sponsorship. Secondly we need to be honest about this, both with the community and ourselves.

]]>Edward Snowden recently provided testimony to the European Parliament's Civil Liberties, Justice and Home Affairs (LIBE) committee as part of its inquiry on electronic mass surveillance of EU citizens.
There's better analysis and comment elsewhere but I thought it's important enough to be available in a readable plain text format,]]>https://www.xavierriley.co.uk/transcript-of-edward-snowdens-testimony-to-eu/8c7331c2-433f-4190-8387-4c34dfca43e9Mon, 10 Mar 2014 12:00:00 GMTEdward Snowden recently provided testimony to the European Parliament's Civil Liberties, Justice and Home Affairs (LIBE) committee as part of its inquiry on electronic mass surveillance of EU citizens.
There's better analysis and comment elsewhere but I thought it's important enough to be available in a readable plain text format, rather than a PDF.

Introductory Statement

I would like to thank the European Parliament for the invitation to provide testimony for your inquiry into the Electronic Mass Surveillance of EU Citizens. The suspicionless surveillance programs of the NSA, GCHQ, and so many others that we learned about over the last year endanger a number of basic rights which, in aggregate, constitute the foundation of liberal societies.

The first principle any inquiry must take into account is that despite extraordinary political pressure to do so, no western government has been able to present evidence showing that such programs are necessary. In the United States, the heads of our spying services once claimed that 54 terrorist attacks had been stopped by mass surveillance, but two independent White House reviews with access to the classified evidence on which this claim was founded concluded it was untrue, as did a Federal Court.

Looking at the US government's reports here is valuable. The most recent of these investigations, performed by the White House's Privacy and Civil Liberties Oversight Board, determined that the mass surveillance program investigated was not only ineffective -- they found it had never stopped even a single imminent terrorist attack -- but that it had no basis in law. In less diplomatic language, they discovered the United States was operating an unlawful mass surveillance program, and the greatest success the program had ever produced was discovering a taxi driver in the United States transferring $8,500 dollars to Somalia in 2007.

After noting that even this unimpressive success -- uncovering evidence of a single unlawful bank transfer -- would have been achieved without bulk collection, the Board recommended that the unlawful mass surveillance program be ended. Unfortunately, we know from press reports that this program is still operating today.

I believe that suspicionless surveillance not only fails to make us safe, but it actually makes us less safe. By squandering precious, limited resources on "collecting it all," we end up with more analysts trying to make sense of harmless political dissent and fewer investigators running down real leads. I believe investing in mass surveillance at the expense of traditional, proven methods can cost lives, and history has shown my concerns are justified.

Despite the extraordinary intrusions of the NSA and EU national governments into private communications world-wide, Umar Farouk Abdulmutallab, the "Underwear Bomber," was allowed to board an airplane traveling from Europe to the United States in 2009. The 290 persons on board were not saved by mass surveillance, but by his own incompetence, when he failed to detonate the device. While even Mutallab's own father warned the US government he was dangerous in November 2009, our resources were tied up monitoring online games and tapping German ministers. That extraordinary tip-off didn't get Mutallab a dedicated US investigator. All we gave him was a US visa.

Nor did the US government's comprehensive monitoring of Americans at home stop the Boston Bombers. Despite the Russians specifically warning us about Tamerlan Tsarnaev, the FBI couldn't do more than a cursory investigation -- although they did plenty of worthless computer-based searching - and failed to discover the plot. 264 people were injured, and 3 died. The resources that could have paid for a real investigation had been spent on monitoring the call records of everyone in America.

This should not have happened. I worked for the United States' Central Intelligence Agency. The National Security Agency. The Defense Intelligence Agency. I love my country, and I believe that spying serves a vital purpose and must continue. And I have risked my life, my family, and my freedom to tell you the truth.

The NSA granted me the authority to monitor communications world-wide using its mass surveillance systems, including within the United States. I have personally targeted individuals using these systems under both the President of the United States' Executive Order 12333 and the US Congress' FAA 702. I know the good and the bad of these systems, and what they can and cannot do, and I am telling you that without getting out of my chair, I could have read the private communications of any member of this committee, as well as any ordinary citizen. I swear under penalty of perjury that this is true.

These are not the capabilities in which free societies invest. Mass surveillance violates our rights, risks our safety, and threatens our way of life.

If even the US government, after determining mass surveillance is unlawful and unnecessary, continues to operate to engage in mass surveillance, we have a problem. I consider the United States Government to be generally responsible, and I hope you will agree with me. Accordingly, this begs the question many legislative bodies implicated in mass surveillance have sought to avoid: if even the US is willing to knowingly violate the rights of billions of innocents -- and I say billions without exaggeration -- for nothing more substantial than a "potential" intelligence advantage that has never materialized, what are other governments going to do?

Whether we like it or not, the international norms of tomorrow are being constructed today, right now, by the work of bodies like this committee. If liberal states decide that the convenience of spies is more valuable than the rights of their citizens, the inevitable result will be states that are both less liberal and less safe.

Thank you.

I will now respond to the submitted questions. Please bear in mind that I will not be disclosing new information about surveillance programs: I will be limiting my testimony to information regarding what responsible media organizations have entered into the public domain. For the record, I also repeat my willingness to provide testimony to the United States Congress, should they decide to consider the issue of unconstitutional mass surveillance.

Rapporteur Claude Moraes MEP, S&D Group

Given the focus of this Inquiry is on the impact of mass surveillance on EU citizens, could you elaborate on the extent of cooperation that exists between the NSA and EU Member States in terms of the transfer and collection of bulk data of EU citizens?

A number of memos from the NSA's Foreign Affairs Directorate have been published in the press.

One of the foremost activities of the NSA's FAD, or Foreign Affairs Division, is to pressure or incentivize EU member states to change their laws to enable mass surveillance. Lawyers from the NSA, as well as the UK's GCHQ, work very hard to search for loopholes in laws and constitutional protections that they can use to justify indiscriminate, dragnet surveillance operations that were at best unwittingly authorized by lawmakers. These efforts to interpret new powers out of vague laws is an intentional strategy to avoid public opposition and lawmakers’ insistence that legal limits be respected, effects the GCHQ internally described in its own documents as "damaging public debate."

In recent public memory, we have seen these FAD "legal guidance" operations occur in both Sweden and the Netherlands, and also faraway New Zealand. Germany was pressured to modify its G-10 law to appease the NSA, and it eroded the rights of German citizens under their constitution. Each of these countries received instruction from the NSA, sometimes under the guise of the US Department of Defense and other bodies, on how to degrade the legal protections of their countries' communications. The ultimate result of the NSA's guidance is that the right of ordinary citizens to be free from unwarranted interference is degraded, and systems of intrusive mass surveillance are being constructed in secret within otherwise liberal states, often without the full awareness of the public.

Once the NSA has successfully subverted or helped repeal legal restrictions against unconstitutional mass surveillance in partner states, it encourages partners to perform “access operations.” Access operations are efforts to gain access to the bulk communications of all major telecommunications providers in their jurisdictions, normally beginning with those that handle the greatest volume of communications. Sometimes the NSA provides consultation, technology, or even the physical hardware itself for partners to "ingest" these massive amounts of data in a manner that allows processing, and it does not take long to access everything. Even in a country the size of the United States, gaining access to the circuits of as few as three companies can provide access to the majority of citizens' communications. In the UK, Verizon, British Telecommunications, Vodafone, Global Crossing, Level 3, Viatel, and Interoute all cooperate with the GCHQ, to include cooperation beyond what is legally required. http://www.theguardian.com/business/2013/aug/02/telecoms-bt-vodafone-cables-gchq

By the time this general process has occurred, it is very difficult for the citizens of a country to protect the privacy of their communications, and it is very easy for the intelligence services of that country to make those communications available to the NSA -- even without having explicitly shared them. The nature of the NSA's "NOFORN," or NO FOREIGN NATIONALS classification, when combined with the fact that the memorandum agreements between NSA and its foreign partners have a standard disclaimer stating they provide no enforceable rights, provides both the NSA with a means of monitoring its partner's citizens without informing the partner, and the partner with a means of plausible deniability.

The result is a European bazaar, where an EU member state like Denmark may give the NSA access to a tapping center on the (unenforceable) condition that NSA doesn't search it for Danes, and Germany may give the NSA access to another on the condition that it doesn't search for Germans. Yet the two tapping sites may be two points on the same cable, so the NSA simply captures the communications of the German citizens as they transit Denmark, and the Danish citizens as they transit Germany, all the while considering it entirely in accordance with their agreements. Ultimately, each EU national government's spy services are independently hawking domestic accesses to the NSA, GCHQ, FRA, and the like without having any awareness of how their individual contribution is enabling the greater patchwork of mass surveillance against ordinary citizens as a whole.

The Parliament should ask the NSA and GCHQ to deny that they monitor the communications of EU citizens, and in the absence of an informative response, I would suggest that the current state of affairs is the inevitable result of subordinating the rights of the voting public to the prerogatives of State Security Bureaus. The surest way for any nation to become subject to unnecessary surveillance is to allow its spies to dictate its policy.

The right to be free unwarranted intrusion into our private effects -- our lives and possessions, our thoughts and communications -- is a human right. It is not granted by national governments and it cannot be revoked by them out of convenience. Just as we do not allow police officers to enter every home to fish around for evidence of undiscovered crimes, we must not allow spies to rummage through our every communication for indications of disfavored activities.

Could you comment on the activities of EU Member States intelligence agencies in these operations and how advanced their capabilities have become in comparison with the NSA?

The best testimony I can provide on this matter without pre-empting the work of journalists is to point to the indications that the NSA not only enables and guides, but shares some mass surveillance systems and technologies with the agencies of EU member states. As it pertains to the issue of mass surveillance, the difference between, for example, the NSA and FRA is not one of technology, but rather funding and manpower. Technology is agnostic of nationality, and the flag on the pole outside of the building makes systems of mass surveillance no more or less effective.

In terms of the mass surveillance programmes already revealed through the press, what proportion of the mass surveillance activities do these programmes account for? Are there many other programmes, undisclosed as of yet, that would impact on EU citizens rights?

There are many other undisclosed programs that would impact EU citizens' rights, but I will leave the public interest determinations as to which of these may be safely disclosed to responsible journalists in coordination with government stakeholders.

Shadow Rapporteur Sophie Int'Veld MEP, ALDE Group

Are there adequate procedures in the NSA for staff to signal wrongdoing?

Unfortunately not. The culture within the US Intelligence Community is such that reporting serious concerns about the legality or propriety of programs is much more likely to result in your being flagged as a troublemaker than to result in substantive reform. We should remember that many of these programs were well known to be problematic to the legal offices of agencies such as the GCHQ and other oversight officials. According to their own documents, the priority of the overseers is not to assure strict compliance with the law and accountability for violations of law, but rather to avoid, and I quote, "damaging public debate," to conceal the fact that for-profit companies have gone "well beyond" what is legally required of them, and to avoid legal review of questionable programs by open courts. (http://www.theguardian.com/uk- news/2013/oct/25/leaked-memos-gchq-mass-surveillance-secret-snowden)

In my personal experience, repeatedly raising concerns about legal and policy matters with my co-workers and superiors resulted in two kinds of responses.

The first were well-meaning but hushed warnings not to "rock the boat," for fear of the sort of retaliation that befell former NSA whistleblowers like Wiebe, Binney, and Drake. All three men reported their concerns through the official, approved process, and all three men were subject to armed raids by the FBI and threats of criminal sanction. Everyone in the Intelligence Community is aware of what happens to people who report concerns about unlawful but authorized operations.

The second were similarly well-meaning but more pointed suggestions, typically from senior officials, that we should let the issue be someone else's problem. Even among the most senior individuals to whom I reported my concerns, no one at NSA could ever recall an instance where an official complaint had resulted in an unlawful program being ended, but there was a unanimous desire to avoid being associated with such a complaint in any form.

Do you feel you had exhausted all avenues before taking the decision to go public?

Yes. I had reported these clearly problematic programs to more than ten distinct officials, none of whom took any action to address them. As an employee of a private company rather than a direct employee of the US government, I was not protected by US whistleblower laws, and I would not have been protected from retaliation and legal sanction for revealing classified information about lawbreaking in accordance with the recommended process.

It is important to remember that this is legal dilemma did not occur by mistake. US whistleblower reform laws were passed as recently as 2012, with the US Whistleblower Protection Enhancement Act, but they specifically chose to exclude Intelligence Agencies from being covered by the statute. President Obama also reformed a key executive Whistleblower regulation with his 2012 Presidential Policy Directive 19, but it exempted Intelligence Community contractors such as myself. The result was that individuals like me were left with no proper channels.

Do you think procedures for whistleblowing have been improved now?

No. There has not yet been any substantive whistleblower reform in the US, and unfortunately my government has taken a number of disproportionate and persecutory actions against me. US government officials have declared me guilty of crimes in advance of any trial, they've called for me to be executed or assassinated in private and openly in the press, they revoked my passport and left me stranded in a foreign transit zone for six weeks, and even used NATO to ground the presidential plane of Evo Morales - the leader of Bolivia - on hearing that I might attempt to seek and enjoy asylum in Latin America.

What is your relationship with the Russian and Chinese authorities, and what are the terms on which you were allowed to stay originally in Hong Kong and now in Russia?

I have no relationship with either government.

Shadow Rapporteur Jan Philipp Albrecht MEP, Greens Group

Could we help you in any way, and do you seek asylum in the EU?

If you want to help me, help me by helping everyone: declare that the indiscriminate, bulk collection of private data by governments is a violation of our rights and must end. What happens to me as a person is less important than what happens to our common rights.

As for asylum, I do seek EU asylum, but I have yet to receive a positive response to the requests I sent to various EU member states. Parliamentarians in the national governments have told me that the US, and I quote, "will not allow" EU partners to offer political asylum to me, which is why the previous resolution on asylum ran into such mysterious opposition. I would welcome any offer of safe passage or permanent asylum, but I recognize that would require an act of extraordinary political courage.

Can you confirm cyber-attacks by the NSA or other intelligence agencies on EU institutions, telecommunications providers such as Belgacom and SWIFT, or any other EU-based companies?

Yes. I don't want to outpace the efforts of journalists, here, but I can confirm that all documents reported thus far are authentic and unmodified, meaning the alleged operations against Belgacom, SWIFT, the EU as an institution, the United Nations, UNICEF, and others based on documents I provided have actually occurred. And I expect similar operations will be revealed in the future that affect many more ordinary citizens.

Shadow Rapporteur Cornelia Ernst MEP, GUE Group

In your view, how far can the surveillance measures you revealed be justified by national security and from your experience is the information being used for economic espionage? What could be done to resolve this?

Surveillance against specific targets, for unquestionable reasons of national security while respecting human rights, is above reproach. Unfortunately, we've seen a growth in untargeted, extremely questionable surveillance for reasons entirely unrelated to national security. Most recently, the Prime Minister of Australia, caught red-handed engaging in the most blatant kind of economic espionage, sought to argue that the price of Indonesian shrimp and clove cigarettes was a "security matter." These are indications of a growing disinterest among governments for ensuring intelligence activities are justified, proportionate, and above all accountable. We should be concerned about the precedent our actions set.

The UK's GCHQ is the prime example of this, due to what they refer to as a "light oversight regime," which is a bureaucratic way of saying their spying activities are less restricted than is proper (http://www.theguardian.com/uk/2013/jun/21/legal-loopholes-gchq-spy-world). Since that light oversight regime was revealed, we have learned that the GCHQ is intercepting and storing unprecedented quantities of ordinary citizens' communications on a constant basis, both within the EU and without http://www.theguardian.com/uk/2013/jun/21/gchq-cables-secret- world-communications-nsa). There is no argument that could convince an open court that such activities were necessary and proportionate, and it is for this reason that such activities are shielded from the review of open courts.

In the United States, we use a secret, rubber-stamp Foreign Intelligence Surveillance Court that only hears arguments from the government. Out of approximately 34,000 government requests over 33 years, the secret court rejected only 11. It should raise serious concerns for this committee, and for society, that the GCHQ's lawyers consider themselves fortunate to avoid the kind of burdensome oversight regime that rejects 11 out of 34,000 requests. If that's what heavy oversight looks like, what, pray tell, does the GCHQ's "light oversight" look like?

Let's explore it. We learned only days ago that the GCHQ compromised a popular Yahoo service to collect images from web cameras inside citizens' homes, and around 10% of these images they take from within people's homes involve nudity or intimate activities (http://www.theguardian.com/world/2014/feb/27/gchq-nsa-webcam-images-internet-yahoo). In the same report, journalists revealed that this sort of webcam data was searchable via the NSA's XKEYSCORE system, which means the GCHQ's "light oversight regime" was used not only to capture bulk data that is clearly of limited intelligence value and most probably violates EU laws, but to then trade that data with foreign services without the knowledge or consent of any country's voting public.

We also learned last year that some of the partners with which the GCHQ was sharing this information, in this example the NSA, had made efforts to use evidence of religious conservatives' association with sexually explicit material of the sort GCHQ was collecting as a grounds for destroying their reputations and discrediting them (http://www.huffingtonpost.com/2013/11/26/nsa-porn-muslimsn4346128.html). The "Release to Five Eyes" classification of this particular report, dated 2012, reveals that the UK government was aware of the NSA's intent to use sexually explicit material in this manner, indicating a deepening and increasingly aggressive partnership. None of these religious conservatives were suspected of involvement in terrorist plots: they were targeted on the basis of their political beliefs and activism, as part of a class the NSA refers to as "radicalizers."

I wonder if any members of this committee have ever advocated a position that the NSA, GCHQ, or even the intelligence services of an EU member state might attempt to construe as "radical"? If you were targeted on the basis of your political beliefs, would you know? If they sought to discredit you on the basis of your private communications, could you discover the culprit and prove it was them? What would be your recourse?

And you are parliamentarians. Try to imagine the impact of such activities against ordinary citizens without power, privilege, or resources. Are these activities necessary, proportionate, and an unquestionable matter of national security?

A few weeks ago we learned the GCHQ has hired scientists to study how to create divisions amongst activists and disfavored political groups, how they attempt to discredit and destroy private businesses, and how they knowingly plant false information to misdirect civil discourse (https://firstlook.org/theintercept/2014/02/24/jtrig-manipulation/).

To directly answer your question, yes, global surveillance capabilities are being used on a daily basis for the purpose of economic espionage. That a major goal of the US Intelligence Community is to produce economic intelligence is the worst kept secret in Washington.

In September, we learned the NSA had successfully targeted and compromised the world's major financial transaction facilitators, such as Visa and SWIFT, which released documents describe as providing "rich personal information," even data that "is not about our targets" (http://www.spiegel.de/international/world/spiegel-exclusive-nsa-spies-on-international-bank- transactions-a-922276.html). Again, these documents are authentic and unmodified - a fact the NSA itself has never once disputed.

But we should be clear these activities are not unique to the NSA or GCHQ. Australia's DSD targeted Sri Mulyani Indrawati, a finance minister and Managing Director of the World Bank (http://www.theguardian.com/world/2013/nov/18/australia-tried-to-monitor-indonesian- presidents-phone). Report after report has revealed targeting of G-8 and G-20 summits. Mass surveillance capabilities have even been used against a climate change summit.

Recently, governments have shifted their talking points from claiming they only use mass surveillance for "national security" purposes to the more nebulous "valid foreign intelligence purposes." I suggest this committee consider that this rhetorical shift is a tacit acknowledgment by governments that they recognize they have crossed beyond the boundaries of justifiable activities. Every country believes its "foreign intelligence purposes" are "valid," but that does not make it so. If we are prepared to condemn the economic spying of our competitors, we must be prepared to do the same of our allies. Lasting peace is founded upon fundamental fairness.

The international community must agree to common standards of behavior, and jointly invest in the development of new technical standards to defend against mass surveillance. We rely on common systems, and the French will not be safe from mass surveillance until Americans, Argentines, and Chinese are as well.

The good news is that there are solutions. The weakness of mass surveillance is that it can very easily be made much more expensive through changes in technical standards: pervasive, end-to-end encryption can quickly make indiscriminate surveillance impossible on a cost- effective basis. The result is that governments are likely to fall back to traditional, targeted surveillance founded upon an individualized suspicion. Governments cannot risk the discovery of their exploits by simply throwing attacks at every "endpoint," or computer processor on the end of a network connection, in the world. Mass surveillance, passive surveillance, relies upon unencrypted or weakly encrypted communications at the global network level.

If there had been better independent and public oversight over the intelligence agencies, do you think this could have prevented this kind of mass surveillance? What conditions would need to be fulfilled, both nationally and internationally?

Yes, better oversight could have prevented the mistakes that brought us to this point, as could an understanding that defense is always more important than offense when it comes to matters of national intelligence. The intentional weakening of the common security standards upon which we all rely is an action taken against the public good.

The oversight of intelligence agencies should always be performed by opposition parties, as under the democratic model, they always have the most to lose under a surveillance state. Additionally, we need better whistleblower protections, and a new commitment to the importance of international asylum. These are important safeguards that protect our collective human rights when the laws of national governments have failed.

European governments, which have traditionally been champions of human rights, should not be intimidated out of standing for the right of asylum against political charges, of which espionage has always been the traditional example. Journalism is not a crime, it is the foundation of free and informed societies, and no nation should look to others to bear the burden of defending its rights.

Shadow Rapporteur Axel Voss MEP, EPP Group

Why did you choose to go public with your information?

Secret laws and secret courts cannot authorize unconstitutional activities by fiat, nor can classification be used to shield an unjustified and embarrassing violation of human rights from democratic accountability. If the mass surveillance of an innocent public is to occur, it should be authorized as the result of an informed debate with the consent of the public, under a framework of laws that the government invites civil society to challenge in open courts.

That our governments are even today unwilling to allow independent review of the secret policies enabling mass surveillance of innocents underlines governments' lack of faith that these programs are lawful, and this provides stronger testimony in favor of the rightfulness of my actions than any words I might write.

Did you exhaust all possibilities before taking the decision to go public?

Yes. I had reported these clearly problematic programs to more than ten distinct officials, none of whom took any action to address them. As an employee of a private company rather than a direct employee of the US government, I was not protected by US whistleblower laws, and I would not have been protected from retaliation and legal sanction for revealing classified information about lawbreaking in accordance with the recommended process.

It is important to remember that this is legal dilemma did not occur by mistake. US whistleblower reform laws were passed as recently as 2012, with the US Whistleblower Protection Enhancement Act, but they specifically chose to exclude Intelligence Agencies from being covered by the statute. President Obama also reformed a key executive Whistleblower regulation with his 2012 Presidential Policy Directive 19, but it exempted Intelligence Community contractors such as myself. The result was that individuals like me were left with no proper channels.

Are you aware that your revelations have the potential to put at risk lives of innocents and hamper efforts in the global fight against terrorism?

Actually, no specific evidence has ever been offered, by any government, that even a single life has been put at risk by the award-winning journalism this question attempts to implicate.

The ongoing revelations about unlawful and improper surveillance are the product of a partnership between the world's leading journalistic outfits and national governments, and if you can show one of the governments consulted on these stories chose not to impede demonstrably fatal information from being published, I invite you to do so. The front page of every newspaper in the world stands open to you.

Did the Russian secret service approach you?

Of course. Even the secret service of Andorra would have approached me, if they had had the chance: that's their job.

But I didn't take any documents with me from Hong Kong, and while I'm sure they were disappointed, it doesn't take long for an intelligence service to realize when they're out of luck. I was also accompanied at all times by an utterly fearless journalist with one of the biggest megaphones in the world, which is the equivalent of Kryptonite for spies. As a consequence, we spent the next 40 days trapped in an airport instead of sleeping on piles of money while waiting for the next parade. But we walked out with heads held high.

I would also add, for the record, that the United States government has repeatedly acknowledged that there is no evidence at all of any relationship between myself and the Russian intelligence service.

Who is currently financing your life?

I am.

Shadow Rapporteur Timothy Kirkhope MEP, ECR Group

You have stated previously that you want the intelligence agencies to be more accountable to citizens, however, why do you feel this accountability does not apply to you? Do you therefore, plan to return to the United States or Europe to face criminal charges and answer questions in an official capacity, and pursue the route as an official whistle-blower?

Respectfully, I remind you that accountability cannot exist without the due process of law, and even Deutsche Welle has written about the well-known gap in US law that deprived me of vital legal protections due to nothing more meaningful than my status as an employee of a private company rather than of the government directly (http://www.dw.de/us-whistleblower-laws-offer- no-protection/a-17391500). Surely no one on the committee believes that the measure of one's political rights should be determined by their employer.

Fortunately, we live in a global, interconnected world where, when national laws fail like this, our international laws provide for another level of accountability, and the asylum process provides a means of due process for individuals who might otherwise be wrongly deprived of it. In the face of the extraordinary campaign of persecution brought against me by my the United States government on account of my political beliefs, which I remind you included the grounding of the President of Bolivia's plane by EU Member States, an increasing number of national governments have agreed that a grant of political asylum is lawful and appropriate.

Polling of public opinion in Europe indicates I am not alone in hoping to see EU governments agree that blowing the whistle on serious wrongdoing should be a protected act.

Do you still plan to release more files, and have you disclosed or been asked to disclose any information regarding the content of these files to Chinese and Russian authorities or any names contained within them?

As stated previously, there are many other undisclosed programs that would impact EU citizens' rights, but I will leave the public interest determinations as to which of these may be safely disclosed to responsible journalists in coordination with government stakeholders. I have not disclosed any information to anyone other than those responsible journalists.