Friday, April 27, 2012

I was planning on continuing my series of blog posts today on Neo4J however I was saddened yesterday but the news of the death of Todd Simko. Todd was once the members of one of my favorite bands (Pure) and also someone whom I had the joy of working with in the studio. This photo was taken the last time we were together with John Webster recording our second studio album at mushroom studios. Todd Simko has a credit on 22nd Century's 2011 "Where's Howie!??" but the words "mastered by Tom Simko" just don't do him justice.

Left to right - John Webster, myself (Duane Chaos) and the late Todd Simko.

I clearly remember this day. We were trying to do the final mixing of the song Runaway and John kept blowing fuses and getting frustrated. The studio runner (Alexis) went out and came back with some fine French wine which we made short work of. I distinctly remember John Webster repeating "keep that stuff away from the mixing console you two" about every five minutes. These were good times. Just guys, hanging out doing music, there was no time, we were all in the present and enjoying that special moment. Todd, may you rest in peace. Your inspiration to so many of us will never die. Thank you for the gifts and memories you gave us both in person and on stage. Your style inspires today and your friendliness is infectious.

Addendum: Todd's wife issued a statement which I will repeat here:

"Minna Simko has offered the following statement on Facebook concerning her husband's passing:

"Todd may your soul be finally at peace. You are not only the most fantastic father and husband one could have, but also you were also a great friend, musician, teacher, guide, mentor and music engineer.

"The music community has suffered now a great loss and talent. Not only did you touch all the hearts of people in music but also in the lives of family, my peers and especially our daughters friends and family. You will forever remain close by in our hearts and you will be greatly missed by all."

Thursday, April 26, 2012

Yesterday I write a brief intro to Neo4J and promised to write more. Yesterday's post focused in on what a Graph Database is. Today we'll actually download and install the software itself. First, and I know you all hate this part, please make sure you have the pre-requisites installed. The people who write most software today all take time to document this so you get to have a better experience.

For reference, the machine I am on is a Mac Pro, 12 GB RAM and running OSX 10.7.3. It has 2 X 3 GHz Quad Core Intel Xeon processors so it should make short work of anything I through at it. My machine is Unix based so please modify the instructions based on your operating system. Linux users will be roughly the same.

2. There are a variety of options open. Unless you want to start with the über - enterprise version or build Neo4J Community from GitHub source, simply grab the latest stable version. In this case it is the Community (ie "free") version 1.7. If you have a slow connection, use the time wisely to view Emil's video while the download completes. You will have to choose a location for the download.

3. Unpack the source with whatever tools your Operating System provides. Once unpacked, it will look something like this:

4. To start Neo4J, it is quite easy. Grab a terminal (Shell) and navigate to the <neo4J_Home_Directory>/bin directory and type in sh ./neo4j This will give you a list of available options for starting the database as shown below:

5. These are very simple and self explanatory. To start neo4J, simply type in sh ./neo4j start

6. There are two ways to verify the neo4j instance has started. the first is to type in the command sh ./neo4j status which gives you a simple acknowledgement that it has started and the process ID (Unix based systems). A second, more verbose set of details, can be retrieved by typing in the command sh ./neo4j info This gives you a wealth of information including every jar in the CLASSPATH, JAVA_OPTS (options), the environmental variable JAVA_HOME, the NEO4J_INSTANCE which is a path to the current instance, the server PORT it is using over HTTP and the NEO4J_HOME environmental variable as well as the current JDK value.

Validating the port is fairly easy. Just grab your handy browser (Chrome please) and go to http://localhost:7474 (unless you've already changed the configuration file to a different port). YOu should see a newly initalized databased as such:

So where can we configure these? Let's start with the JAVA_OPTS. YOu will see a line that looks like this -Dorg.neo4j.server.properties=conf/neo4j-server.properties The first part of this configuration starting with -Dorg.neo... is specified within a file that is under the /conf directory named neo4j-wrapper.conf. The JVM parameters are all specified here but are actually pointers to other configuration files. If you open this fine, you will see the same lines here:

Note that on OSX, the Java Version simply says "CurrentJDK" which is pretty bloody useless. If you really want to know the JDK version, use your command window and type in java -version. In my case I am running

java version "1.6.0_31"

Java(TM) SE Runtime Environment (build 1.6.0_31-b04-415-11M3635)

Java HotSpot(TM) 64-Bit Server VM (build 20.6-b01-415, mixed mode)

Within the neo4j-wrapper.conf there are several properties such as the initial and maximum java heap sizes along with some warnings. It is probably a good idea to become familiar with the warnings before you go about tinkering with these settings. The one I found useful was to be able to uncomment line 10 to allow garbage collection logging. This data can be valuable in determining what is going on under the hood so I change it with every install.

This information is important for the next steps when we will embed neo4J into an Eclispe project. For today, let's play with our new Neo4J instance. In order to get the neo4j shell, simply type in the command (under the same <neo4J_Home_Directory>/bin directory) sh ./neo4j-shell This will now enable you to look around at some of the available options. Type in help for a list of commands.

The shell commands are all well documented at http://docs.neo4j.org/chunked/stable/shell-starting.html. Note that the shell is configured and enabled from the configuration of the Neo4j kernel (again in the /conf folder). Okay - enough foreplay. Let's make some nodes! Build your first few nodes ny using the mknode command. I can make two nodes, one with my name and one with my wife's name as such.

neo4j-sh (0)$ mknode Duane

neo4j-sh (0)$ mknode Bettina

If I go back and check the browser aimed at http://localhost:7474 it will confirm that now in addition to the root node, there are two additional nodes for a total of three. Congratulations. You have just made some nodes! So how do you get to those nodes? This is why I started the documentation on Technoracle. I found it confusing using the neo4j-shell since almost every command resulted in an empty query until I read the documentation (an engineer's last resort). The GraphDB works almost exactly like a unix filesystem which means your learning curve should be tens times faster (assuming you're familiar with unix commands). When you invoke the shell, you are basically in the "~" directory or "me" as the neo4J folks call it. To traverse somewhere (to Duane or Bettina for example, you need to make a relationship using the mkrel command. It is very simple. Type in the following,

mkrel -ct KNOWS Duane

Now let's dissect this. mkrel is the command to make a relationship. There are two variables "-ct". C should be supplied if you are creating a new node (the wording is a bit rough to read using the "man pages and it took a while to figure out that a a relationship is basically a node as well.

If you want to nuke the graph, you can also do this using rmnode to delete nodes and rmrel to delete relationships. rmnode comes with a nasty little flag -f which has the same effect as the unix command "su rm -r *" which removes everything. As you probably guessed, using rmnode with the -f flag was irresistable and I had to try it so I typed in rmnode -f. Now beware, this removes all including the current node. Once you run this and try an ls, the return will be a question mark since there is no current node (or at least it didn't seem to be reachable. After running this you get:

neo4j-sh (?)$ ls

Node <ref> not found

To fix this, simply create a new root current node. You can even use JSON to give it a more robust meaningful name.

neo4j-sh (?)$ mknode --cd --np "{'name':'me'}"

neo4j-sh (me,10)$

Don't ask me why I do stuff like this. I think I just like to explore what is possible and how to recover before doing any serious work.

Earlier we created nodes that had no relationships we could traverse. Now we will use a different syntax to create new nodes we can reach. This easier way to create new nodes you can traverse to using the shell is to use the mkrel command. This command can actually create the new nodes as well as the relationship between the current and newly created node. To do this, type in the following:

mkrel -t LOVES neo4j and then type in ls into the prompt after that has finished.

Ta-da! You can now traverse the node using the trav command. The traverse command is very complex and can build very powerful statements and filters. For this lesson, all we want to do is go from the current node to the node we just created (node 11) in this case. To do this, use the syntax

trav -o depth -r LOVES:both,HAS_.*:incoming

trav = traverse

-o depth = the traversal order. The only possible values are BREADTH_FIRST DEPTH_FIRST breadth or depth. Think of these as controls over a funnel - very wide and shallow or very narrow and deep)

-r LOVES:both,HAS_.*:incoming= the r flag sets the relationship type. In this case apparently (me) - [:LOVES] -> (11) or I love node 11. To be honest, I might delete it right after this tutorial as I am already growing tired of this relationship ;-) I am somewhat not clear on the ":both,HAS_.*:incoming" however I believe it specifies that the traversal is to disregard the fact it is incoming or outgoing.

Okay - enough for today boys and girls. More soon. In the meantime, please do try this at home!

Wednesday, April 25, 2012

I've worked with databases for a long time. Recently, a came across neo4J and cannot believe how awesome it is. I want to devote this blog post to helping people get it installed and the fun you can have.

First, a little bit abut Graph Databases. Graph databases are substantially different than RDBMS systems. As one person puts it, if you write, you can write code, if you can draw you can draw graphs. It is really that simple. A graph database starts with a root node. The database is comprised of nodes, relationships, indexes and properties. A simplification of this is the following chart:

A simple graph database.

A graph database keeps track of nodes and relationships as well as indexes (we'll get into that later). For now think of this in kinderSpiele terms. A graph is simply a drawing that show how things are connected. Each connection might have a name. each node might also have a name. Here is a simplistic view of a graph.

In this simple graph, Duane - [LOVES] -> Neo4j, the latter of this has a property of being binary. Now psychology aside (this in fact would be an unhealthy physical relationship), this captures several important concepts yet leaves out several very relevant ontological answers. Relationships organize Nodes into structures that allow a Graph to resemble many natural structures including a List, a Tree, a Map, or a compound Entity – any of which can be combined into yet more complex, richly inter-connected structures. It is obvious that Duane loves Neo4J but here are some questions that are left unanswered.

1. Does neo4J love Duane back?

2. Is Neo4J even aware that Duane loves it?

3. Is Duane able to see that Neo4J is in fact a binary node and probably not suited for a proper relationship?

All are possible but undefined in this scenario. That is why the next concept that must be introduced is a traversal mechanism. Traversals allow navigation of graphs via statements that can select exact routing between many of these objects. THese can be written in many languages such as cypher and allow a filter to be applied to find a path though the nodes and relationships to find answers to certain questions. Such a question in the real world may be "How many friends do I have who enjoy eating spumonte ice crean while reading up on graph databases on Technoracle". In reality that subset of the population is likely very small but when applied to something like Facebook or Google Plus, become highly relevant.

Here is a depiction of how traversals work. Again this is rudimentary.

Again,. this is very simple but you get the idea. The traversal mechanism can take a set of instructions, then use if to find data it requires very efficiently. An example might be that you use Facebook. When you log in, it starts with the node of "you". As the page loads, the javaScript on the page creates a backend query that says find all the nodes that are related to the user down to a layer of X deep. Neo4J's Java API supports depth limits making it idea for this sort of operation. Unlike an RDBSM system where an entire table might have to be walked, Neo4J allows you to set limits and take actions based on the current state. Paths are predefined statements, often written in Cypher.

Keeping with the Facebook example, an INDEX is often a useful tool. When a certain node is required a a start point over and over again, you can use it as an index to start with. By contract, RDBS systems use a table and rows lookup to find the startpoint. The index is simply a contextual based starting point. Indexes can map directly to a node, a relationship or backwards from a property. Instead of saying:

SELECT * FROM TABLES WHERE * EQUALS "Duane Nickull".... you can tell a graphDB to "get Duane Nickull" then traverse outwards from him. Simple and efficient.

Neo4J is a commercially supported, free and open source graph database that is going to rock the world. Trust me on this. Next post will be getting started. All the sordid details (at least 3 easy steps) it takes to get up and running.

Tuesday, April 24, 2012

Yesterday I made a post on the unfair and immoral practice of using a company like SonicBids that gets a monopoly on a music event causing all acts that want to play there to "PayP" to apply to play at the event. They get exclusive rights on events like SXSW. I found to my horror that they were trying to move into Canada via an event at the Renaissance hotel here. I worded a polite but factual post to help them understand the system they were promoting. It got posted but was censored by the hotel chain webmaster. here is the link to the event:

Monday, April 23, 2012

As both a tech guy and a musician, I have made a decision to boycott SXSW. They use an evil and monopolistic business practice to force bands to PAY MONEY to apply to play music using a service called Sonicbids. If you are a musician, your trade (performing music) is your primary source of income hence it is your "work". One year we (22nd Century) decided to apply to play given we had fairly good radio audience in Texas and also had lots of MySpace and other fans in the area. The tech/developer crowd is also one of our primary audiences. We went to the SXSW website and it read something to the effect of:

"All artists interested in performing at SXSW must complete an online application through our exclusive online musical event submission platform Sonicbids." followed by the sentence "You'll only hear from us before that time if we encounter a problem or require more information.Source: http://sxsw.com/music/shows/faq

To apply to work your trade, indie artists are forced to hand over $40.00. The facts can be verified at this web page on Sonicbids. Once you apply (yes we did this once even though it is morally offensive), it is not transparent. You apply along with thousands of other artists and hear nothing back. That's it. They took your money and you got nothing in return. There is no clear statement about how many SonicBids artists actually get to play at SXSW. It says they are exclusive, but I find it hard to believe guys like Duff McKagan, a former music panelist at SXSW and good friends with my friends Randy Rampage and Zippy Pinhead, would pay this fee. Is it possible that SXSW hires the bands they want anyways and maybe gives X remaining slots to SonicBids hires? No one knows the truth other than SonicBids and SXSW.

All of a sudden you see an announcement like this one from NPR music saying that Fiona Apple, Bruce Springsteen and others are playing. Let me ask you some basic questions.

Do you think they (Bruce and Fiona) paid $40.00 to apply to work at SXSW? Would you pay your boss $40.00 every day to ask him if you could work and the response might either be "yes" or complete silence?

I personally answer "no" to both of these questions. Read the statement above again. It says "exclusive". Here is an open question to anyone. Have you ever seen a band play at SXSW through another vehicle that SonicBids? If so, please tell us about it.

So what is going on here? Simple. This is illegal in most countries. We are musicians. Granted I do not need to personally make money on music but many of my dear friends to. This system is shameful and it is not conducive of fair trade. SXSW should immediately to the right thing and move to a crowd sourced model which musicians are allowed to freely enter and the system is completely transparent. This closed doors, money grab is a slap in the face to the artists, many of who are trying to simply survive while pursuing their passion in music. Who loses? Everyone except SonicBids. You the music fans at SXSW do not get to vote on who you want to see, the musicians lose money and get noting in return and SXSW makes the ultimate choices. SXSW does not get the best customer experience from those who attend, hence also loses. I am not sure if SonicBids kicks back any money to SXSW and would love to see the financial statements. Oh, I guess they are not for public eyes.

"Who's benefiting from it? All the wrong people," stresses Jeff Price, founder and CEO of TuneCore, a digital music distribution service. "The traditional music industry now has a new income stream that is based on other people's music and copyrights being exploited and sold. It is the largest global scam that exists in the music industry today, depriving artists and songwriters of hundreds of millions of dollars."

Jeff nailed it (thank you Jeff). Not only are the artists getting their music downloaded for free, they are forced to pay to apply to ply their trade, and the streams are also stolen and no money flows back. This is morally offensive. I write this blog post and ask you all to think about this. I'm sure if you're planning to go to SXSW you will still go but if there is anything you can do to raise attention to this situation, my brothers and sisters, the artists of the music industry, the lowest rung on the ladder, would appreciate it. You see, we no longer have power in this matter. This is a plea to you to help us.

It's very simple. If we do not have fair trade introduced into the mix, the arts suffer. It becomes run by the mainstream media who decide for you what you will see. All we ask for is an equal chance to perform. All we ask is to be able to ply our trade without being robbed, used and actually get a fair compensation for our work. We put in ten thousand hours of practice, buy our own instruments and gear, pay our own way to go to shows and perform. Being preyed on by monopolistic business practices hurts us.

Dear SonicBids, if you ever try to come to British Columbia, you will be met with a swift response of getting the BC labour relations board immediately aware that you are violating several provincial statues around the labour market. That is before the BC Gaming Commission looks at you for opening up a "lottery" and allowing bands with minors to place bets based on their believe they will be hired to money. If you do manage to get by them, any gig you try in a major city will be met with swift and violent boycotts. I will put myself on the front line as many of my Vancouver friends will do to stop you from migrating your business to British Columbia, Canada. We will not let any musician play a gig in Vancouver if they had to pay SonicBids to apply to play! It is that simple. Stay out of our province. There several hundred of us who will show up at such a gig to shut it down.

Dear people elsewhere, use this as a template. Understand the system and realize by going to SXSW you are part of a system that takes advantage of indie musicians and artists, who are often the amongst the poorest of the system. Help support them by demanding SXSW and other bars engage in fair trade practices and demand your right to crowd source music. Bring this up at panels or ask the people on the music panel what they think about this. Raise awareness. Help us!

To SXSW - expect me to be a thorn in your side until you fix this. I will personally offer to help you devise and implement a better system that treats musicians fairly. If you show the courage to acknowledge this error and move forward to correct the situation, I am positive the crowds of people attending will be happy to help vote the bands they want to see play. The ball is in your court to acknowledge an injustice and correct it. Until then, you will not see me and I will be vocal in fighting what I and my local laws deem illegal.

SXSW - the next move is yours to fix this. I'm here to help if you want.

Pin it!

SHARE!

If you are helped by this blog in any way, or just find the contents interesting, please give it a share to help keep me publishing. Shares help drive my advertising revenue which makes me more likely to write articles to help people in the future. Thank you in advance!