Archives By Category

Archives By Month

Subscribe

I’ve been studying Clojure quite a bit this last month. I really love the language and want to use it but I think I am going to be stuck with PHP because of deployment issues and there being too much of a learning curve.

PHP is probably the worst programming language I have come across. But, I find myself asking the question, then why am I programming in it almost exclusively for my “real work”? My day job requires me to program in PHP and then any clients that I do freelance work for require me to write in PHP as well.

I noticed this pattern has happened to me when I was learning Ruby on Rails. I spent months learning it but then when I tried to use it in “the real world” I found that I couldn’t. My clients were using shared hosting. If I developed any products I could only use it on my own servers and couldn’t sell it to others. There would be no way for them to get it up and running without being sysadmins.

Clojure had the idea of using the JVM so that it has access to the entire Java API. This is a brilliant idea but unfortunately I don’t live in a Java world. Everyone I know and all my work opportunities are in the LAMP world.

I could use Clojure and just offer SaaS solutions but for selling actual products people can download and install on their server, PHP is the only option.

The other problem is if I did develop some stuff in Clojure and then needed the same functionality in PHP I would have to write it twice.

So I have a few choices as I see it.

First, I can continue learning Clojure and in another month or 2 I should be proficient enough to be developing web applications in it. I can concentrate on SaaS only and ignore the market for people being able to buy software they can install on their own servers.

Second, I could just stick to PHP and deal with it. There is no learning curve, I already know the language and can crank out work immediately (although probably not at the same pace as I could once I learned Clojure). This is probably the most pragmatic solution but I’m not so sure it would make me happy and there is no leverage involved in it. I could create something good but not great.

Third, What about programming Lisp but targeting the LAMP stack? I see 2 ways of going about this. I could start with Lisp code and then either have a Lisp interpreter in PHP or compile the Lisp into PHP. I did some searching on google and I have not found anyone doing something similar unfortunately. That means I would have to do it myself.

I am leaning towards option three. I have done some work with language parsing in PHP before for a Forth based language in PHP. I should be able to do a Lisp version as well.

Interpreted would be the easiest but would also be really slow. PHP is the slowest language out there and running an interpreter on top of that would be even slower. Still, it should be enough and with prices coming down and hardware improving it really does make business sense. Most of the time is going to be spent in the database doing queries and there is always caching.

Later on I might be able to figure out a way to compile the Lisp into PHP code.

When you’ve been programming for so long sometimes it is hard to take a step back and rethink the way you are doing things. As developers of technologies, languages and platforms we tend to build things incrementally as there is a need. This usually works to some extent but in some regards we end up with a horribly inefficient development experience.

Let’s take the current web development experience on the LAMP stack. One of the first things that strikes me as wrong is the sheer number of languages we have to deal with for any moderately complex project. We have PHP, MySQL, Javascript, HTML and CSS. Are you fucking kidding me? That’s 5 radically different languages while working on the same thing. Worst yet, I’ve seen code that uses all 5 of those languages within a few lines of code. Let’s not forget all the extra work we need to do for cross browser compatibility (always having to write to the least common denominator, or worst yet doing things multiple times for each different browser).

Other times we use one language to output the code to another language for the same data. <?php echo “var myVar=’$myVar’; ?> just so Javascript can use something from PHP. How retarded is that?

What do any of those languages do that would make it impossible or undesirable to do all of that in 1 language? Absolutely nothing! Now, I’m not saying they should all be written in the same style; quite the opposite in fact; but with domain specific languages (DSLs) it is possible to express radically different domains and concepts within a single language. LISP and Ruby both have good implementations for large portions of this but I think it can be taken much further.

Having one language for everything allows us to reduce huge amounts of complexity. Ever have to escape escaped characters? You end up having to type “\\\\” just to get “\”. Having 1 language also makes for better integration between all the parts of the web application. Why do we have to write a validator in Javascript and then do the exact same thing, in an entirely different language (PHP), for the server side when it does exactly the same thing? If we throw Actionscript / Flex into the mix we have even more overhead and complexity.

Then of course we have languages that are pushed to do things they really weren’t designed to do. Ever try to center an element vertically on a page with CSS?

Why is it that 70% of the code we write is plumbing to move data from one place to another instead of logic or algorithms? We have an HTML form that submits the data to PHP, which then needs to read those data and store them into variables, which then need to have PHP dynamically generate SQL that contain those data to be submitted to a database. We have the exact same problem going the other direction.

Does this seem a bit out of whack to anyone?

Let’s use some zero based thinking and transport ourselves 50 years into the past. Let’s say someone tasked us with creating something akin to the current Internet. How would we do it? I highly doubt it would look anything like what we currently have. If we had to build everything from scratch we most likely would end up with something dramatically better.

An analogy we can use is a giant snowball that is rolling downhill. It keeps getting bigger and bigger. Nobody wants to stand in front of it and try to slow it down for fear of getting squashed. It seems hopeless like there is nothing we can do. I don’t think this is the case. In fact, look at the cost if we don’t. I have no doubt if the entire system were redesigned the mid-level programmer could see a 10x increase in productivity. Do we really want to be working at 1/10th of what we could be doing for the rest of our lives?

I have some thoughts on how things can be done different which I hope to expand upon in future essays.

In the meanwhile I hope my words have sparked a curiosity in your mind as to how things can be done differently. I welcome your thoughts and opinions.

In my search of various langauges I have stumbled upon Clojure. It meets a lot of the criteria established in the last post.

I am ashamed to say I have not learned a Lisp dialect up until this point. I have known enough about Lisp to get the concept but never actually used it for anything. The fact that it isn’t commonly available on servers nor common to set up caused me to shy away from it. The thought process going on in my mind was probably: Why bother learning something that I’m not really going to be able to use? It’s just an intellectual curiosity. Well, now I have good reason to learn it.

I bought the Pragmatic book Programming Clojure and like what I have learned so far. This language / environment has a lot of what I have been looking for.

It runs in the JVM so you have full access to any library that is available in Java. Moreover, it is often more convenient to access these libraries in Clojure than they are in Java. It can be embedded into any server architecture that runs the JVM. I haven’t looked into this step but I imagine it is pretty easy to get up and running on a server.

It has all the benefits of Lisp like homoiconicity and macros.

It takes great steps towards Intentional Programming. Since code is data you are not restricted to working with source code in some kind of lexical form. You can use tools that will output the code for you and compile it into JVM bytecodes on the fly and behind the scenes.

There’s much more that can be elaborated on but I think you get the picture. I definitely recommend checking the language out. You can find out more at the web site www.clojure.org.

I have been giving some thoughts on what an ideal language would be like and have come up with the following list.

Flexibility. There is no one ideal language. Design, like many other fields, has always been about deciding between 2 alternatives. What is good in one instance may be horrible in another. Therefore, flexibility is important to make the language fit to the task. This means there are mulitple ways of doing things. It also means you can build the language up to the task. Lisp and Forth based languages are commonly programmed in this manner. To achieve this goal I believe the language will need to be circular and homoiconic.

Simplicity. The most elegant solutions are usually the simpliest as well. This could be thought of as the RISC of programming languages. Again, languages like Lisp are known for their simplicity. There is very little syntax to learn and yet it is very expressive and powerful.

Support for meta programming. I’m not sure if this should be built into the language or if the flexibility of a language will automatically allow for it. The latter would be more elegant. Often when programming I find myself doing the same thing over and over again. So called design patterns are often just artifacts of a language that doesn’t allow you to express your intent succintly.

Homoiconic. This basically means the language data structures of the language are the language itself. Lisp and Factor have this in common.

Multi-fix Syntax. Postfix, prefix, infix should be allowed whenever it is most convenient to do so. Each programming task may be written better in one of these forms. The language should not be restricted to only 1 of them.

Dynamic Syntax. This goes along with flexibility. You should be able to configure the langugae / environment that best fits the task at hand. Perhaps this means lots of languages in one (all of them homoiconic and sharing the same code contsructs internally).

Intentional Programming. I’m not going to define what this is here. You can google it to find out more about it. The basic idea I want to borrow from it is that programming should not be about text only code. You should be able to use tools and construct tools to create what you need.

Macros. This may or may not be needed depending on how usable the langugae is already. I need to spend more time working with Lisp to really understand how they are used and if they should be adapted to my language.

Interpreted. There is tremendous flexibility and power when a language is interpreted. Many things are simplified. Speed and code security are the obvious drawbacks.

Runs inside common environments. PHP, Javascript, Actionscript, and Java are the languages of the web. If the environment is built to run inside these environments you gain all the API from them. It is also much easier to deploy and distribute to others if you have a kernel that can easily be installed without any sysadmin work. This will allow products to be sold without the need of complex environment requirements.

Mix environment with code. I’m thinking of Squeak, the Factor listener, and Intentional Programming here. Programming should not be about working with text only. Often this is the only way it is done. There is no reason why it needs to be that way. There is much to be gained if we allow an environment to directly access the constructs of the language directly.

There hasn’t been a paradigm shift in software development for quite some time. I think the main reason for this is because of our myopic tendency to assume software development can only happen with programming languages based off some lexical syntax. Visual Basic and the following rapid application development (RAD) IDE’s have shown us this is not true for some aspects of code.

I wish to explore Intentional Programming and talk about some of the new ideas it presents. You can read the link for more details but basically Intentional Programming gets rid of the lexical code portion and allows the programmer or user direct access to the parse tree. Lisp allows you to program the parse tree directly but Intentional Programming is a little bit different. It says, yes, the program is eventually a parse tree but give the end user or programmer tools to build the parse tree for each particular problem domain. Often times the “intention” of the programmer is easier expressed graphically, or maybe by recording a macro, or maybe by drawing lines between objects representing data objects. For example, if you needed to read 5 columns from a database table, it would make much more sense to pop up that table in a GUI and then just click on each column. From there it would store that “intention” in the parse tree. Only the data and the intention would be stored. The intention can be thought of as a function that operates on the data. So instead of writing everything using code in a text editor, you use a lot of small tools to construct a parse tree in the most natural and intuitive means possible for each and every problem domain.

Why is it that we are only using text in such a limited fashion? We are not using color, size or position to represent any additional syntatic meaning. This makes for inefficient communication between the programmer and the computer. It would be easier to read code if color, size, or position could be used to represent more. This is readily seen when using color syntax highlighting in an editor. It is much easier to scan code to understand the meaning. The human brain can spot things quicker that it would have otherwise skipped over.

Why are we using only text and not graphics to represent our code? The compiler takes some kind of input and translates that into machine code. Why does it always have to be text. Why can’t we use something that is easier to read and write code in? I’m not exactly sure what this might be. Perhaps it could be some kind of flow chart, or it could be wiring objects together like Legos.

I don’t claim to have all answers to these questions. I am throwing them out there in the hopes that it triggers imagination in others. I intend to share my thoughts on these questions as I have insights.

Programming in it’s essential nature is breaking down a problem and then specifying the solution to that problem in a way that a computer can repeat. Why do we always use the same language most of the time? We tend to have our favorite languages and we use them to solve our problems. This greatly limits our ability to think about the problem. We break the problem down into the components of the particular language we want to use. This goes back to the saying, if all you have is a hammer, everything looks like a nail. Programmers are often unaware there is a better way of solving their problem because they think they only have the tools the language provides them.

I think a better way to go about solving problems is to have a generic framework for building our solutions and then builidng the language to suit the problem. This goes back to intentional programming. A programmer should not just use what is in front of him to solve a problem but build the tools he needs to solve the problem as well. This is top-down and bottom-up development at the same time.

Frequently programmers develop the same kinds of things over and over. It is to the advantage of the programmer to learn more about building tools, especially about building his own small languages to express a problem in a manner that is concise and without needless syntatical artifacts.

I recommend programmers take a look into tools like ANTLR to get a better idea of what I am talking about and to expand their minds.

ANTLR in and of itself will not allow you to combine multiple languages together. This is why I said some kind of general universal framework is needed. I think intentional programming has a lot of ideas that can help advance us in that direction.

I will attepmt to cover these ideas more in future posts. I apologize if my writing is difficult to follow or lacking in clarity or explanation. This is my first time trying to write my thoughts down in any serious manner. Hopefully, my writing will improve with each post.

In the last few years I have become increasingly aware of a chasm between programmers and computer scientists. It seems like the two have nothing in common when in fact they should be one and the same. To be a truly great programmer you must also be a great computer scientist. Why is it that so many programmers have no concept of computer science?

This chasm has been there all the time, but has been made more apparent the more I study computer science. What is the difference between a programmer and a computer scientist you ask?

This post stereotypes the programmer and computer scientist. I exaggerate the weaknesses in each to illustrate a point. Obviously, this is not necessarily the case and this post should not be viewed as a rant or an attack on the two types of individuals mentioned here.

The Programmer

The typical programmer is someone who writes code on a daily basis. They are familiar with a few programming languages and their API’s. The vast majority of programmers are only aware of the C family of languages (C, Javascript, Java, PHP, Actionscript, etc) as opposed to functional languages (LISP, Scheme, Erlang) or concatenative languages (Forth, Factor). They typically write the same type of code over and over and never think they can abstract it out to make their work easier and less repetitive. Their thinking, and hence their ability to solve a problem, is constrained by the tools they use. Even after doing something over and over they never have the thought, “There must be an easier way to do this”.

The Computer Scientist

The computer scientist is someone who knows a lot of theory, is well versed in math, and understands how computers work at a fundamental level. They see the computer as an abstract system of computation. They are usually more involved with academia rather than day to day programming. When confronted with a problem they break it down into its fundamental parts and then build a model to solve it.

In short, the computer scientist is someone who sees a problem and builds a tool to solve it. The programmer is someone who is familiar with a few tools (languages / API’s) and uses them to solve his problem.

The Problem

The problem is the programmer is constrained by his tools. You may have heard the saying, “If all you have is a hammer, everything looks like a nail”. The programmer will eventually get the job done but not necessarily in the most elegant or efficient manner.

The computer scientist doesn’t really spend much time in the trenches, so to speak, to know the problems facing the average programmer. They are more concerned with solving purely intellectual problems. As a result the computer scientist doesn’t build the tools the programmer needs or does so in a way that is not very useful, convenient or easy to use for the programmer.

So, computer scientists don’t actually use the tools they create or know what tools need to be created and programmers only know about the typical tools other programmers are familiar with. Programmers don’t spend much time researching to see what other tools are out there or bother to create their own tools.

The Reason

There are some reasons why these situations exist.

The programmer usually writes code for other people. They are charged with getting a job done and they are only interested in getting that job done as quickly as possible. When confronted with a problem they look at their tools they have in front of them and then set out to work on the problem. As soon as they get the job done there is more work for them to do. As a result the programmer typically has little time to learn new languages, environments or API’s.

Not having a solid background or interest in computer science the programmer often does not know there is a better way of doing things. They have learned from other programmers that is all there is. They are not accustomed to taking a step back, abstracting out what they are doing, and developing a system that allows them to get the job done faster in the future.

Building tools takes time. If it takes 30 minutes to get a task done and it looks like it would take a full day to create a system to do the same job in 30 seconds the typical programmer is not going to want to spend their time working on it. They are not going to want to explain to their boss why what previously took 30 minutes is now taking 8 hours. If there are other programmers around, they don’t want to look like an incompetent idiot while their fellow programmers are, at that moment, finishing much more work.

There is also a break even point. If it takes 30 minutes to solve the problem and the problem only comes up a few times a year it would be counter-productive for the programmer to spend 8 hours coming up with a system to let him solve it in 30 seconds in the future. If on the other hand, it is something the programmer does on a daily basis, that same 8 hours is going to pay for itself in less than a month.

That’s a 6000% ROI in a month. How would you like to make that in the stock market? Unfortunately, the typical boss is even less aware of a better way of doing things or is in the same situation where he is reporting to someone else and doesn’t want to have to explain why it is taking longer than normal.

Typically you are not going to see 60 times increase from creating a tool. But you may see 5 times the increase and that’s nothing to ignore.

As mentioned before, the computer scientist lives in the world of academia. They typically don’t work on the same things a programmer would on a daily basis. As a result the problems they solve are a little grander in scale and do not always have direct application. Again, this is a generalization to illustrate a point. Academia has provided enormous value to modern programming.

The Solution

I have attempted to illustrate the weaknesses of strong polarization to only one side. The obvious solution is to move towards the middle. The programmer should learn more about computer science and theory. The computer scientist should spend more time working on real world projects.

For programmers, I recommend the books The Pragmatic Programmer and Code Generation. Study other programming languages, especially those that are radically different from what you are used to. LISP, Ruby, and Factor are all good candidates I would recommend. Look at open source projects, especially frameworks and see how they do things. They will expand your mind as to what is possible and make you much more efficient.

Code Innovator is dedicated to innovating how code is written. This a copy of what I wrote on the about page.

Motivation

For quite some time I have had this nagging feeling that the way we currently write code is woefully inefficient. I feel this sense of vision and calling that compels me to say “There is a better way”. At the time of this writing I only have a few scattered ideas and vague concepts of what we can do better. But, part of me just knows there is a better way. It’s almost clairvoyant in nature. It’s like it’s on the tip of my tongue and I’m struggling to communicate it. I can’t describe it. This site will document what I am able to communicate at the moment and my odyssey to further manifest this vision.

By researching and studying other programming languages I will learn new programming idioms that will change the way I program and teach me to better use the languages I already know. When a writer expands their vocabulary they are able to express their thoughts more eloquently. They are able to use less words to communicate the same idea and they are familiar with more concepts. It is my hope that increasing my “vocabulary” of programming idioms will allow me to design a better language.

Unanswered Questions

Programming languages haven’t really changed in the last 30 years. It seemed like in the beginning, when people first started writing software, there were all kinds of innovations. What happened? Why has it almost completely stopped?

Why is it that we can explain to a programmer exactly what we want in a few minutes and yet it will take that same programmer hours, if not days, to write it? Sometimes it is because computers are just not capable of understanding what we want. Other times the computer may lack the neccesary domain knowledge to solve the problem. Frequently though, this is not the case, or at least it doesn’t need to be the case. Often, it is because there is an insufficient level of abstraction. Closing the gap between intention and implementation will be one of the main topics of focus.

The Plan

I intend to take a structured approach to solving this problem.

define the problem / pain

define the possible goals

research other languages and development methodologies to discover the general characteristics of their strengths and weaknesses

implement ideas

experiment and test the ideas with real world applications

report

refine / repeat process

I will cover this in another post.

About the Author

My name is Brennan Cheung and I have been programming for a little over 20 years. I started programming computers at the age of 7 on the Commodore 64. My primary interest at that time was to learn how to write video games. I started off with Basic and them assembly on the C64. Later I moved onto the IBM PC, C, x86 assembly, Java, and then a whole bunch of other languages after that.

I started off with video game programming, then in high school I started to gain interest in operating systems. Right now the bulk of my programming is web based programming to one degree or another. Improving this area is my primary intent.

I am currently employed full time as a programmer. I enjoy spending my free time learning about compilers and different languages. I also enjoy glamour photography and have had my work published in various magazines, web sites, and calendars.