Posted
by
michaelon Friday June 13, 2003 @06:24PM
from the bingo-eureka dept.

Pete Bevin writes "Artima has a fine interview with James Gosling, creator of Java, about his latest project. It's called Jackpot, and it treats the parse tree as the program. This makes refactoring much, much more intuitive. The article has some good insights into code visualization, technical writing, and making your programs more understandable."

So you can write what is kind of like a reverse grammar, where you associate structural patterns with what you can think of almost as TeX descriptions of how to represent the patterns graphically. What you see on the screen has been generated from this pattern matching. So we can, on a user chosen basis, turn various program structures into all kinds of visual representations.

I don't know much Java, but.NET has an entire CodeDOM namespace that can be used to generate assemblies and code on the fly. DOM being the keyword - it presents C# code as a parsed object tree. I haven't played with it beyond generating simple assemblies but I wonder if it could be somehow cajoled into creating a tree representation that also understands flow. That would be a neat thing to play around with.

Yes, and do you realize the significance of Gosling Emacs? It was its proprietary nature. Gosling Emacs was what finally got GNU Emacs off the ground as a successful open source Emacs implementation for UNIX systems.

Gosling has been hostile to open source from day one. He created NeWS and tried to kill off X11, and he was probably partially behind keeping Java proprietary.

un-fsking-believable. Someone posts opinion of java being then emacs and he gets modded into flamebait ?/. editors : please revoke this flamebait moderator of his moderation rights. IU have never used emacs NOR vi, but how can u possibly call something flamebait if one compares two projects of teh same author !

Maybe in another 10 or 20 years ALL programming languages will be as powerful as Lisp.

Actually, the other languages do make progress in that direction. Look, for example, at Python: It has objects, metaclasses, introspection, an interactive environment, functions as data (even if lambdas could've been done better), strong dynamic typing, runtime extensibility (albeit clumsy), not to mention small conveniences like keyword and rest args or docstrings.

Oh, absolutely. I agree 100%. My sarcasm intentionally was based on truth.

Basically what I was saying (which is certainly not my own discovery) is that the semantics of Lisp/Scheme/Other Lisp Dialects move "language features" into the realm of the developer (as opposed to the language designers). Plus there is the bonus of being able to operate on the program itself. And for these reasons, pretty much any feature somone is planning to ad

As a Slashdot thread on a programming language progresses, the probability of someone claiming that "Lisp already does that" approaches unity.

If I had mod points I'd mark you up as funny - but have you read and grokked the Meta Object Protocol ?? Because much as I hate Lisp at the lower syntactic levels, I keep on finding that features I like in other languages were actually present in the MOP and similar. That's not to say that other languages don't present the ideas in better and easier-to-use ways, but it still pisses me off that those beardie-weirdie Lisp blokes had already thought of it so much earlier...

I've heard that somewhere before...
Oh yeah. In my ANSI Common Lisp book. Something about the real power of Lisp being that everything, including the program itself is just a tree structure.

Well, actually, there seems to be more to Jackpot's methods of code visualization than that. Lisp code can be thought of as having a tree-like structure, but it may not be as clear as what Jackpot's visual representation my be. What Jackpot would do is show the annotated parse tree, so it can give much information about how it is constructed. It would be a useful graphical representation that appears to go beyond what Lisp code would show, and with that representation and the source code, you can get the best of both worlds.

Anyway, they also mention that you can implement a "reverse grammar" that would take data formed in parse trees and make code more readable. For example, you can have Greek letters and other mathematical notation such as the square root symbol. If you have long equations in your program, this could be very useful in making your code readable, and thus understandable.

So what Jackpot seems to be is a way of giving different ways of viewing the code you write, which, IMHO, can go a long way in solving problems with it and simply improving on it.

Yes, yes, I had to deal with all the "Lisp did it first" comments when Eidola [eidola.org] was on Slashdot.

While it's true that the program is the parse tree in Lisp, that's not a very strong statement. Lisp's elegance comes from the fact that there are so few constructs in the language, and basically everything is a list -- even your programs. But they're basically just lists, that's all. So you have this wonderful flexibility, but the parse tree doesn't actually tell you very much about the program; you have to "parse the parse tree" to recognize higher-level constructs.

Now languages with lots of language-level constructs -- like strong static types, objects, access modifiers, etc. -- tell you a whole lot about high-level structure with their parse trees. (And, for those following along at home, Lisp is not such a language -- not that that's a bad thing, but it isn't. Lisp builds these high-level constructs out of a very few language-level atoms.) To my knowledge, applying the "language is the parse tree" principle to non-functional languages is still largely the domain of research projects like Jackpot, Eidola, and Intentional Programming, and visual languages.

Moral: Lisp is very, very, very cool, but it has not already done everything every other language is doing. So yes, it may sound familiar from you Lisp book, but it's not the same.

Well static typing makes the parse tree richer in the same way that requiring you to list your religion on a driver's license application makes the DMV's database richer. I'm not convinced of the benefit.

It (the former) tells you the set of messages (or methods, whatever you wanna call them) that it is legal to send to the value in question. It tells you what kind of thing the value is supposed to be. How can that not be useful in analysis?

Well static typing makes the parse tree richer in the same way that requiring you to list your religion on a driver's license application makes the DMV's database richer. I'm not convinced of the benefit.

It (the former) tells you the set of messages (or methods, whatever you wanna call them) that it is legal to send to the value in question. It tells you what kind of thing the value is supposed to be. How can that not be useful in analysis?

I would dispute that. All you have to do is prove that each part works as specced and that no interactions throw the whole thing out of whack. In other words, "unit proving" and "integration proving", akin to unit testing and integration testing.

"It (the former) tells you the set of messages (or methods, whatever you wanna call them) that it is legal to send to the value in question. It tells you what kind of thing the value is supposed to be. How can that not be useful in analysis?"

Well, it doesn't help any in a language where anything and everything can be done at run-time. Most problems that benefit from this type of analysis are hard, and most hard problems are very dynamic. Besides which, what benefit would this have for analysis beyond the

But they're basically just lists, that's all. So you have this wonderful flexibility, but the parse tree doesn't actually tell you very much about the program; you have to "parse the parse tree" to recognize higher-level constructs.

No. You can parse the parse tree to reason about or modify programs. You can as well use higher-level constructs, for example asking for the class of an object and manipulate it, the declared types of variables and functions, etc. The whole metaobject protocol is about giving you an object-oriented interface to your program internals, and the same style shows in various other places. Basically, a lot of what is lost at compile-time in most other languages is a live first-class object in Lisp - for a simple example, you can get the package of a symbol, see what other packages it imports, change its name, and make some symbols in it not being exported.

Moral: Lisp is very, very, very cool, but it has not already done everything every other language is doing. So yes, it may sound familiar from you Lisp book, but it's not the same.

Indeed. But Lisp is the only language so far that allowed adding new concepts in portable ways, without having to modify the underlying implementation. CLOS, the object system, is basically a bunch of functions and macros, and if you don't like its class/generic-function based approach, just load a package that implements a prototype-based one and use that. An implementation of Eiffel-style design by contract is about two screenful of code, adding final and abstract classes is less.

Are these additions "language-level constructs"?
Hard to tell. The syntax the programmer deals with is just as if it were, even if everything eventually gets expressed in lower-level terms. The distinction is just not meaningful in Lisp - there just is no hard barrier between the language designer and the user, Lisp users design their language all the time.

But Lisp is the only language so far that allowed adding new concepts in portable ways, without having to modify the underlying implementation. CLOS, the object system, is basically a bunch of functions and macros, and if you don't like its class/generic-function based approach, just load a package that implements a prototype-based one and use that.

Well, yeah, and that's just the thing -- because it's so wonderfully extensible, your ability to reason programmatically about a program is limited. To analyz

real power of Lisp being that everything, including the program itself is just a tree structure.

Lisp 1.5 in someways wasn't Lisp 2.0 because the original form of the language was in terms of "M-expressions" that looked like mathematical expressions and Fortran. The idea was to have a compiler that translated the M-expressions into the internall forms of S-expressions.
However as John McCarthy Says [stanford.edu]

The project of defining M-expressions precisely and compiling them or at least translating them into S-exp

I was thinking the same thing. What's the difference between this and LISP or XSLT? I don't think this guy is an inventor of anything. A reinventor and possibly a brilliant marketer, yes. An inventor, I don't think so.

"Complexity is in many ways just evil. Complexity makes things harder to understand, harder to build, harder to debug, harder to evolve, harder to just about everything." -- Gosling

Software entities are more complex for their size than perhaps any other human construct because no two parts are alike. If they are, we make the two similar parts into a subroutine - - open or closed. In this respect, software systems differ profoundly from computers, buildings, or automobiles, where repeated elements abound. -- Fred Brooks, Jr.

Which quote tells you more ? Which quote has more insight ? Which quote came 30 years earlier ?

Here's a clue - complexity in software doesn't usually vanish at some magical point, we just aim to achieve a position where our view of inherent complexity in a problem becomes optimally manageable. As the fundamental point of interest within a problem domain changes over time, so will the optimal viewpoint. The point of re-factoring is to move our viewpoint according to what we want to do now, not what we wanted to do when the code was written.

Gosling is talking techno-babble... tell him to draw a parse tree of any meaning in his jargon.

Hardly. What he's saying there is (to expand a little), when code becomes too complicated, it becomes hard to understand, and thus people waste a lot of time trying (and often failing at first) to debug, evolve etc.

Obviously, he skipped a word or two there, making his statement less than 100% accurate - as any normal human being is wont to do in ordinary conversation - a fact which many overly critical Slashdotters are prone to ignoring!

Hardly. What he's saying there is (to expand a little), when code becomes too complicated, it becomes hard to understand, and thus people waste a lot of time trying (and often failing at first) to debug, evolve etc.

I meant the article was techno-babble. The selected quote (that made me and others laugh) was just plain banal: "complexity is complex" - gee whiz, James, why not say "good is good and bad is bad" - the point is that complexity the fundamental problem we're attempting to manage. If that's as de

The quoted sentence was just dumb. The rest of the article was techno-babble. And I'm sorry I haven't convinced you of my right to comment on it, but I stand by my statement.

Neither is the quote dumb nor is the article techno-babble.

The article does not ask you to agree in any way with Goslin. The article tells you what Goslin is working on currently. If you dont find his work exciting then there are basicly 2 reasons possible: a) you are not interested in that research area. b) you did not understand

The quote from Goslins article/interview IMHO tries to say that computer programs are often more complex coded then the problem they try to solve requires.

And where does it say that ? Nowhere - you're taking something that you know and reading it into a quote that doesn't say it. Judge the article not by what you already think, but by what it says, and you'll find it says nothing, but flatters the reader. Now take Brook's argument [gsu.edu] about the essential and accidental complexity of software (that I alluded

Hardly. What he's saying there is (to expand a little), when code becomes too complicated, it becomes hard to understand, and thus people waste a lot of time trying (and often failing at first) to debug, evolve etc.

Why does code become too complicated in Java? Because the language was designed poorly and lacks good mechanisms for abstraction.

If Gosling were concerned with "complexity" or knew anything about how to manage it, he would have designed a better language. Jackpot is not a solution to Java's

Huh? I find the Java API to be incredibly well documented, and usually very intuitive when you need to learn new parts of it. It's only complex if you are trying to learn it in its entirety, and it's not really complex at all compared to the overwhelming amount of stuff you need to do in many programs that it will do for you.

I challenge you to program in Java for a year or so then go back to C++, Python, or pretty much anything else. It may feel great to stretch your legs out and exercise some syntacti

I think they will fire him for such thoughts. But he shouldn't worry. Active State may consider to extend him a job offer. After all he'd rather join the process of Python's rapid evolving than fixing bugs in JVM-beast.

Isn't that what IntelliJ does already? I use it as my main IDE, and it has an amazing understanding of Java -- it allows you to refactor just about anything.When working in it, you feel like you're not just editing Java -- you're editing the fully integrated structure of your software.

Exactly. I know several people who gave up emacs for it. One guy has always hated IDEs with a passion, but once he saw the refactorings and ability to easily browse the code, he made the switch and has never looked back. He still hates IDEs, but he loves IntelliJ IDEA:-)

No, it is not free nor is it open source. You do, however, get what you pay for. The cost is a factor, though, so it may not be for you.

It's supremely configurable, very keyboard friendly, and has more and better features than its rivals, and runs faster with a better UI.

I personally have not used Eclipse, so I cannot make the judgement myself, but I've read threads debating the merits of the two, and IntelliJ's IDEA seems to come out ahead in the end.

Personally, I like the ease of use and intuitiveness. You don't need to work 'its way', but the more you use it, the more features you learn about, and many times the reward is big enough to warent changing long-time habits. One example:

Type out the word 'new' and the first few letters of the class you want. Then hit the 'code completion' key, and then hit 'introduce variable'. Wham. It completes the entire line, declaring the variable in line. You save a vast amount of typing. Or you can 'program by intention'... just start typing code. If you type a method name or variable that doesn't yet exist, just hit Alt-Enter and it'll let you create it. A simple 'back' button and you're back where you were to continue coding.

I could go on forever. Plus, if there's anything you think is missing, or don't like, you can easily write a plug-in to add it or modify the behavior. Many people have. IntelliJ even has a WIKI site for users to post their own plug-ins. They also have an excellent bug reporting system, and they're very responsive.

IntelliJ IDEA has the abbreviations feature you mention, and includes the ability to "surround selected text" with the abbreviation, and expand into complex structures (like for loops with bodies) that even substitute appropriate variable names and types for you (letting you edit them easily after the expansion). It's very powerful.

IntelliJ IDEA doesn't explicitly have the Alt-K/L feature, but it's code completion 'learns' and will put your lengthy java i

Some of this is reminiscent of things the functional programming language folks have been doing for a while. In particular the parse tree concept sounds a bit like graph reduction [brighton.ac.uk], which runs programs by repeatedly simplifying the graph which it (where the graph is a generalization of a parse tree). One of the things you can do with such a system is "common subexpression elimination" where common subtrees are moved to single point, ensuring they're evaluated only once. Sounds like a specialized form of refactoring, doesn't it...?

Of course all this is easier in functional languages, because you don't have to worry about state, identical trees will always evaluate to the same value. Not so in Java, if any of the nodes refer to global data.

I wonder how often Gosling talks with Guy Steele, who was pivotal in the development of both Scheme and Java. I'd love to see what they'd come up with if they put their brains together.

Have you read an article on Java.sun.com lately? Sun makes those kinds of arguments wrt their shipping technologies all the fricken time. Not in an overly pushy way IMO like some companies, and not in a moronic vapid way like those MS ads in Dr. Dobbs Journal (Do MS think that Dr Dobbs readers are complete morons??) But they do. It's refreshing to read an article with a complete lack of such, even if I can't follow it all.

Sun probably does not want to market a tech which may not go into production for yea

You get cleaner, easier to maintain, and easier to debug code, and better multi-threaded scaling simply by avoiding J2EE and some of Sun's other libraries.

As for "merging apps", that's almost always a stupid idea to begin with. The parts of apps that were designed to be reusable should already be in identifiable libraries. The parts of apps that weren't designed to be reusable shouldn't be merged. If they do need to interact, have them communicate via IPC, RPC, DOs, or some database.

When you googled for something *java* it was anoying enough to deal with stupid $tarbuck$links, with 'Jackpot' an unlucky click and it might take 20 min to undo the popdowns, offers for the *best internet casino*... and oh my gawd, add to the wrong mail list!

(I was going to add an example link but I wouldn't do that to my worst enemy)

Alex Stepanov famously
described Java [stlport.org] as "a money-oriented programming language".
I guess that makes the name "Jackpot" an appropriate name.
I suppose the next projects will be "Jingle" and "Jyp".

Hmm... most of what he talks about has been available in proper languages for decades! I'll even try, as a public service, not to mention LISP:]

Smalltalk has this:

(3/5) class methodDictionary at: #+

which will give you the + method for the Fraction class. From that object (the CompiledMethod) you can get the original source code (if it's available), or a parse tree. The parse tree will give you indices into the original source, in case you should need to do a GUI for your IDE:)

For me as along time programmer this feels like moving from C to C++. You can discuss your program on a whole other abstraction level.
GREAT.
Things are getting very interesting in the field of improving the coding process. I still remember Sniff C++ started by Eric Gamma in the early 90-ties. This was the first product to visualize/navigate over large C++ projects that blew me away. One can certainly see this approach in Eclipse - one of the latest projects of Eric Gamma.
A few very smart IDEs appeared - CodeGuide, Eclipse, IDEA.
Eric Gamma was working in an IT research lab in the UBS in Zurich in Switzerland before he became really famouns with his GoF book.
Lars

You can discuss your program on a whole other abstraction level.GREAT.

If you used a better programming language, you wouldn't need complicated tools to "discuss your program on a whole other abstraction level".

The stuff Gamma has been doing is the equivalent of "structured programming for assembly language programmers". It's a set of tricks to make programming in OOLs slightly less awful than it would be otherwise. It is certainly not a great advance in software engineering.

Not a great deal there yet, but if you're interested in Jackpot then the Jackpot home page [sun.com] would be worth bookmarking for future reference. Their early work on source code metrics is interesting and the published papers listed are a good starting point for more detailed information than can be delivered in an interview.

Besides Lisp and Eclipse and IntelliJ, this sort of syntax-tree-oriented manipulation has been going on in optimization and programming language research for the last 20 years. (And Eclipse does a fine job -- all that chit-chat about encapsulating public instance variables is just a choice on the refactoring menu, and it's free. I've got no idea why Gosling thinks this is new and interesting.)

For example, people (including me) at Rice University worked on a source-to-source Fortran vectorizer that manipu

Besides Lisp and Eclipse and IntelliJ, this sort of syntax-tree-oriented manipulation has been going on in optimization and programming language research for the last 20 years. (And Eclipse does a fine job -- all that chit-chat about encapsulating public instance variables is just a choice on the refactoring menu, and it's free. I've got no idea why Gosling thinks this is new and interesting.)

Eclipse does only 10 of over 100 actual "named" and "known" refactorings.

One piece of analysis, therefore, is to find all the public instance variables. But we can find them and also make them private, add all the setters and getters, and account for what it means to actually access the variables via setters and getters.

As I've said before, Gosling appears to have oblivious to most of the research that had gone on in the field of object-oriented languages.

Java should been designed from the start to enforce getter/setter access to instance variables. This fe

While this is certainly possible, I doubt it seriously. NeWS was specifically set up to be able to use OO techniques. I also remember that early discussions and descriptions of java mentioned most of the OO languages of interest at the time and discussed their good and bad points.
I don't know that Dylan was mentioned, but I'd be very surprised given Gosling's background that he was unaware of it. And I think it would be just as hard for him to b

Java should been designed from the start to enforce getter/setter access to instance variables.

Yes. In my dream world, somebody at Sun will get a fucking clue and make getter/setter stuff implicit based on keywords when you declare the variables. Then if you need to override the default getter/setter behavior, you just explicitly declare the methods.

Then we can make the call implicit, throwing out all the foo.getBar().getBaz() crap and just make it foo.bar.baz.

I can't seem to find any details at the moment, but if I remember correctly (quite unlikely, come to think of it) there was a project called "Juice" presented as an alternative to java back in java's early days that represented programs as parse trees. It was developed by Nicklaus Wirth and was an offshoot of oberon.

And there are the obvious advantages that might be gained by representing programs as XML trees (internally at least, programmers don't need to see it).

Most of this stuff isn't new. The first general refactoring tool I ever saw was the RefactoringBrowser built by John Brant and Don Roberts at the University of Illinois at Urbana-Champaign. It uses Smalltalk parse trees to analyze code, perform refactorings, lint checks and code rewrites. It's free and has been incorporated into VisualWorks Smalltalk, Dolphin Smalltalk and Squeak, and is available as an add-on for VisualAge Smalltalk. To see the kinds of refactorings supported, check out: http://st-www

INTERLISP, Warren Teitelman's variant of LISP used at Xerox PARC, had this from about 1978.

INTERLISP's internal representation was the LISP s-expression. Programs lived in huge saved binary workspaces rather than text files, at least while being developed. This had pluses and minuses, especially on the underpowered machines of the period.

There were a number of transformations one could perform on code in this form, and they were guaranteed not to change the semantics of the program. You could designa

2) From the interview: For instance, if you've got a static method that takes an object as a parameter, and it modifies that object, then somebody probably just slapped that method in there because it was easy.

I don't think that's really true. Classes with static methods seem to be very convenient to me. Much like the decorator pattern, where code (here in the static method) only deals with modifying data objects given to it. That

... in 1987. I worked for Xerox AI Systems at the time and was one of the developers for the Lyric release of LOOPS.

LOOPS had:

A single-inheritance object system with GUI support (class, method, and object browsers).

Editing with structure editors that manipulated the parse tree directly. The structure editor was also used as the inspector in the debugger.

refactoring support in the browsers (select a method and move it to another class, etc).

automated global refactoring based on code analysis - this being 1987, it had a pseudo-natural language interface (EDIT ALL METHODS CALLING FOO AND REFERENCING *BAR*...).

LOOPS is one of the primary predecessors of CLOS (the other being Flavors).

Parse-tree-based editing has been around for awhile - Google for "syntax directed editing". Paradoxically, it works much better in Lisp than in other, more syntax-heavy domains; when your editor insists that everything be syntactically and semantically well-formed all the time, it's best that there be very few, very general syntax rules. This is why it also works in Smalltalk.

Gosling has been talking about this for at least two years. I'd really like to see it actually happen, because it's got tremendous potential.

His talk about complexity is a bit facetious; if you really wanted a simple programming language you'd do it all with S and K (trivial functional operators). The trick is not to minimize complexity but to achieve exactly the correct level of complexity.

Object orientation is one way of dealing with complexity: you group like functionality together into boxes with li

When you construct forwarding methods, they're different from the original methods. You can't just replace all uses of the forwarding method by uses of the moved method, because they actually behave slightly differently.

That sounds like a language design problem to me. The solution to that should be to fix the language, not to construct elaborate theorem provers and visual representations of the source code. Building tools to deal with a messy language will just mean that the language and software writt

Hey this is my first post. I have angry half-baked opinions tho.
I find that commenting for a retarded 6 year old makes code unreadable. Have you ever tried to find anything in the apache tomcat sourcecode using only notepad? Why is there a comment on "getInfo()" that says "Returns the info."???
Some classes I was trying to understand I just went through and deleted every comment so I could read the damn thing. Turns out the class was only like 12 lines of code.
It was like shaving a cat.

I think I'll write an SMTP server in Jackpot and advertise it as such. Spambots will assume it's a honeypot, and my users will thank me.

Or I could just deny incoming connections from everyone that isn't large enough to file suit like AOL is doing. That way, I wouldn't have to learn yet another language or edit exim's source to say it's Jackpot.

This type of thing has been thought of before, such as Intentional Programming at Microsoft Research (now at intentsoft.com) or aspect programming at PARC

I don't know much about intentional programming, but how is aspect oriented programming related to this? At its most basic, AOP is mostly just a form of semantic-aware macros (though you can do it without the language support, much like you can to object oriented programming without language support).

Your opinions about Python are all well and fine but if you have the choice between Java and Python on a project, use Java.

Even though I happen to agree that Python is a better language (for the most part), I would always stick to Java. I chose Python on a project I did. I would much rather have that experience on my resume as Java instead. Python is nice, but employers just aren't looking for it.

Sad, but true.

On a side note: Microsoft Site Server 3.0 was written in Python by the original shop that wr

Even though I happen to agree that Python is a better language (for the most part), I would always stick to Java. I chose Python on a project I did. I would much rather have that experience on my resume as Java instead. Python is nice, but employers just aren't looking for it.

Sad, but true.

Python is nice, but employers just aren't looking for it? That's a bad reason, if any...JUST LIE!!!! Does it really matter? What _really_ matters is doing it right and doing so efficiently. If python gives you an a

That's a bad reason, if any...JUST LIE!!!! Does it really matter? What _really_ matters is doing it right and doing so efficiently.

Bzzzt! Wrong answer!

What *really* matters (to me, anyway) is staying employed. Your strategy employs two faulty tactics: lying and not focusing on what the customer needs or say they need. If you lie about your experience, then be prepared to accept the consequences when you're found out. If you do a project in Python when a customer specifically requests Java, then be pr

4. Agreed. Which is why most people don't use Java for the UI. Most UIs are written in VB or Delphi, or web-based (in Flash, or HTML/Javascript).

I encourage you to take a look at IntelliJ IDEA. It blows the lid off the lie of 'show GUIs'. Other than it's start-up time (during which it's parsing your files, building caches, and other time-consuming tasks), it's quite fast. The whole thing is written in Java.

Is it? I've found Swing to be very handy for cross-platform GUI development. Combine it with Java Web Start and you got an easy way to develop and deploy all sorts of nifty stuff. Like this [infoether.com] and this [infoether.com].