Monday, November 06, 2006

We've had a lot of discussions recently about quality management of plone.org/products - how do people who don't spend their days on plone-user and #plone work out what products are worth betting on, where they are going and whether they'll stay viable in the future?

A lot of really great ideas came out of the session we held at the Plone Conference. We've taken the ones that seemed most workable and turned them into a todo-list for the PloneSoftwareCenter.

Friday, October 06, 2006

Since getting back into doing some Java/J2EE web development, I'm constantly struck by the sheer number of mega-frameworks that I feel I probably should know. It's overwhelming, it gives me a nervous twitch, like I could be investing lots of time in some framework that turns out to be inferior because I have no way of comprehending what else is out there.

How do Java web app developers ever get anything done?

Here's a list of frameworks and APIs that I've come across in my current project, all of which could have some relevance to my project:

Java Servlet API

JSP syntax

JSTL

JSF

MyFaces/Tomahawk JSF components

Oracle ADF JSF components

Java Facelets

Hibernate

iBATIS

EJB 2

EJB 3

JBoss SEAM

Spring

Struts

WebWork

SiteMesh

Tapestry

AppFuse

Log4J

JUnit

JMock

JDBC

DBUnit

Cactus

Maven

Cargo

Ant

Tomcat

JBoss

Eclipse

MyEclipse

Oh ... and I need to determine which ones of these can work together, and which of my tools will tolerate which technologies. Aaaaah!

To be clear, my application is basically just forms that drive SQL queries. I don't even care so much about security. However, a goal was to try and use best-practice patterns and technologies, to ensure the application could scale up in the future.

Now, I'm pretty good developer, I know Java (the language) well and I learn quickly. But several of the pieces above would take me a week or more (each) to learn properly. Obviously some are easier than others, and many are directly competing. But how do I know what to choose? How do I know where to invest my time?

I looked for books, but it seems that many of them are out of date by the time they hit the printers. In the blogsphere, there are people who seem to be able to pick and choose between all of these. Power to them, but there is not a great consensus on what is best this week.

If there is some canonical book that covers an end-to-end process with best practices - please, I'd like to know. :)

But I started thinking about how this works in the Plone community. Im sure new users feel a bit overwhelemed, and that we could do things to make it easier for them. However, based on own experiences from when I first played with Plone, it wasn't nearly so bad:

Plone is there, you can touch it and feel it and play with things to see how they break.

Python is terse and readable.

Although Java kicks our ass on API docs, it's easier to follow the actual code since the .py files are all there and legible.

I learn best by example. Like most people, I started with a few copies of some page templates and later some Archetypes classes. That meant I could get results very quickly.

Of course, I'm comparing apples to oranges here. Plone is a CMS and a platform for CMS-like systems. J2EE is much wider in scope. However, my application is not so unique - I need forms, page flow logic and database queries. Maybe one day some CRUD logic, and a few pages of text.

Obviously, this is all in the terriotory of that infamous Better Web App video, and the cliche exodus of Java coders to Ruby-on-Rails. Fair enough, but I'd like to think that it's not impossible to be productive and feel in control of a Java-based solution. This is often a question of skills (people learn Java in school), of prior investment (people have been trained in Java already), of back-end systems (that already run in Java), and of trust (for business-critical components that may feasibly need a web front-end). I refuse to believe that just because Java is strongly typed and compiled, this can't be done right. So most likely I just haven't discovered that framework yet.

However, there is another argument - Java is great for the middle tier, but less productive for the front end. Perhaps we should put more time into finding a bearable way of defining service layers that would let you build your user management and GUI in Plone or Django or Rails and let actual business processes take place in Java beans or EJBs. At least I think that separation would make a lot of decision makers more comfortable - Hibernate, EJB, Oracle (or Postgres) on the back-end, sexy GUI app on the front end that is infinitely more dispsable and lower risk, and is more likely to be what users want if it is written in a framework that encourages agile and rapid development. It may even help to ensure that the middle/back-end business objects are completely decoupled from the GUI and facilitate SOA.

Now, I don't have the time to read a few books on the subject, so I'm feeling a bit ignorant and inadequate. That feeling, however, is amplified many-fold by the fact that I am unable to step through most of the code that's not behaving the way I expected. Perhaps I could get there by downloading big tarballs and configuring source attachments in Eclipse, but I worry this may be a big black hole and I'm reluctant. The balance of JAR files being managed in my IDE and deployed in various cases is already quite precarious, so I don't want to upset it.

The second thing I miss desperately is pdb's (the python debugger) interactive prompt. Some things are definitely not working the way I expect. I can get some way with the Eclipse debugger and watch expressions, but it's cumbersome and slow.

Thirdly, my productivity is hampered by the constant re-compiles. Every time I change something, even a template, I'm recompiling and re-deploying to my local Tomcat instance. It's possible that I could find a way to edit templates in-place in the exploded .war file in Tomcat, or hot-replace them at least, but the only way I knew to do that was with MyEclipse and it doesn't like my Java Facelets templates.

This is not made better by the fact that we have three dependent projects. I have to a dance in Eclipse and Maven that involves 'mvn clean install' on a dependent project, cleaning code in Eclipse, and then re-deploying the project. I just lost nearly an hour pulling my hair out writing banal tests when it turned out that Tomcat wasn't executing the code that was being shown in my debugger. Meanwhile, this blog post has formed whilst I'm waiting for things to re-compile. Each time, my distraction period grows a bit longer, and my productivity dips a bit further.

Lastly (for now) there is way, way too much magic going on with JSF components. As far as I can tell, every form button is actually managed by a generated snippet of inline JavaScript. This (or something else) is managing some state that persists across Tomcat restarts, which sometimes is persisting the component tree (i.e. the tree of JSF components, which is formed by XML tags in various nanespaces, interleaved in non-obvious ways with the DOM of page itself). Sometimes, I get bizarre errors because some remnant of the page from half an hour ago is still being processed. Sometimes, not reliably reproduceable. The only way I've found to reliably test my code is to close down my browser completely and then re-open it after each re-compile/server re-start. Joy.

Further, sometimes the state appears to be saved but it's not - when I click a certain button which has a client-side handler (which is hooked up to a method on a bean by way of some magic expressions) but which aims to stay at the current page after processing, the UI shows form controls with values still entered, but they are magically lost when I click another form button causing an empty form to be submitted. Oh - and sometimes the first command link (think an <a> tag, except it's actually handled with some inline JavaScript on an onclick event and bound into the JSF dispatch machinery) that I click immediately after the application is started leads to a blank page being rendered (something about a phase of the component tree rendering not being ready, I read somewhere), and I have to re-load it to see anything.

I'm sure I'm just ignorant and that there are better ways of going about all of this, but I've spent as much time as I can afford with books and Google, and now I just need to get some work done. I'm patching together working practices that feel inferior and clumsy, but basically work. Luckily, this application is small, so there's not that much to do. But it doesn't feel particularly elegant. (/me wants a view and a few adapters)

Please refrain from pointing and laughing, but if you have some useful tips, I'm all ears.

Monday, September 25, 2006

I really should know better than to post this. Really. Please, don't start a big flame war. But, I peer over the fence from time to time skim through some tutorials and demos on Ruby and Ruby-on-Rails in particular, and one of the things that puts me off it (aside from the sunk cost of my Python knowledge and the fact that my favourite open source CMS project uses Python), is that ... it's so ... ugly.

I have an in-built aversion to blocks that end with an end keyword. Call it prejudice. I once learnt Visual Basic. Maybe that's it, but there are also lots of strange symbols flying about. $`? < for inherits-from? Functions without braces (oh, that's just crazy...).

If someone has seen the beauty in Ruby, please enlighten me. I genuinely believe that code asthetics are important in motivating programmers, ensuring consistency, catching mistakes and ultimately producing better software.

Of course this is totally subjective. But some good examplse of elegance and aesthetics would be nice.

Sunday, September 24, 2006

One of the interesting fallouts from the recent discussions about Nuxeo moving CPS over to Java in the future, is whether relying on tools like IDEs is legitimate, and indeed whether this is just excuse-making for the deficiencies of a language.

I'd argue that it doesn't matter. Whether you call it a language or a tool is largely moot. We don't edit Python in ed, we use vim or Emacs or TextMate (sensible people use the latter). Some languages are complemented well by tools, others have less need for them, and some just don't have very good tools available. The important metric is how efficient you are within your working environment, taken as a whole to include your language, your framework and your full toolchain.

Therefore, the learning curve = the language + the tool. If getting the tools set up is going to take you a long time, you lose productivity. Eclipse is fairly easy going. Some of the plug-ins are a pain to get and install, others are a breeze. But move down one level, and it can become a pain to manage builds with Maven (which is very powerful but can also be very complex when you step outside the box a little) or Ant (which is not that complicated, but requires a fair bit of manual work). I know companies that invest two weeks of a team's time just to get the build environment set up and configured right, which surely is wrong.

And secondly - when the tool does the wrong thing, you probably still need to understand what's going on. That complexity that the tool was hiding is rarely hidden completely, forever, because at some point your tool doesn't yet support the thing you need to do. And in many cases, the tool then expects to manage your source code in its own way and either stomps on your changes or breaks utterly.

Good tool design, it seems, is not much easier than good language design. There are still some things I miss in my current, very minimal Python set-up (consisting of Python, Terminal.app, TextMate, Firefox, FireBug), such as code-completion and support for refactoring by finding and modifying code that needs changing because I renamed a function or moved it up or down the inheritance hierachy. But in most cases, Python is high-level enough that the tools I need are in the language itself, saving me from having to learn a whole new tool.

And in the meantime, I just love doing this:

$ cd Products$ mate CMFPlone CMFCore CMFDefault ATContentTypes

This opens TextMate with all those directories in a project, and I can grep (Cmd+Shift+F), edit, read, svn (Ctrl+Shift+A), add and remove files and do everything else I need. The alternative is to start Eclipse, find the right workspace, and add the particular folder to a project (or open an existing one). Thanks, but by the time Eclipse has loaded, I'm already onto my next task. :)

Thursday, September 21, 2006

I'm going to argue out of ignorance. I am not nearly as close to the Nuxeo guys as I am to many other Zope people, and I don't have very much direct experience with CPS at all. However, I want to speculate a bit about the recent announcement that Nuxeo 5 (which is basically the evolution from CPS 4) will be 100% J2EE, in light of some realities that are abundantly clear at my existing employer.

CPS always felt a bit reclusive in the Zope world. Florent, Leonnart, Jean-Marc and Tariq (I probably spelled half of those wrong) were visible in the Zope 2, CMF and Five spheres, and I really like them and their work. However, CPS itself was (is) one of things that we often thought, "oh, it's really powerful and good" but never really used. Maybe it's because I don't speak French. Maybe it's because I'm addicted to the Plone community. But CPS was (is) Nuxeo's baby and business model. Which is perfectly, perfectly fine (and it's still fully open source, as will Nuxeo 5 be). Florent and Leonnart in particular contributed tons to CMF, Five and other parts of the stack. But it's these contributions we'll miss (presuming they stop, which I hope they don't), not the CPS-specific Zope technology that we never much re-used (or got around to looking at, perhaps).

I've been doing some J2EE work lately. There are things that are quite good about it, and there are things that are not. The Nuxeo 5 FAQ says that Java can be pleasurable with the right tool support. Maybe, but I can count on all my hands and toes the number of hours I've lost to trying to get the build system to play ball, configuring various tools and plug-ins, and debugging strange errors in non-readable code. I'm sure it'd be much better if I worked with Java day in, day out, got tons of training and had it all laid out for me by a standard build that some soul had spent months setting up and testing. At my employer, that'll never be the reality.

I have some particular beef with JSF (Java Server Faces). It may be the best Java's got for web UI, but it's still painfully slow and often unpredictable (I see some component tree state caches that don't invalidate properly, for example). It does not play well with JSP at all. Java Facelets help, and even have METAL-style macros, but don't even think of showing a complex JSF-based UI to a web designer. It's hard to know where the line between HTML and JSF components go, and JSF components still involve writing HTML to an output stream with Java code (they say they're working on it). I'm sure I just don't understand it, but I have seen supposedly cutting-edge JSF work (in Alfresco), and it's still awkard (single templates having a myriad of responsibilities).

The reason that I don't understand it because the stack is a mile high. JSF builds on JSP builds on servlets builds on streams builds on ... EJBs are just scary. Hibernate is quite good, actually, but I found it hard to debug. I'm quite quick at picking up new technologies, but the amount of technologies I have to understand to be productive is so large. I really wanted to use Spring, but that was another book to read. Again, if I did it day in day out and could affort to take a month on a training course or two, I'd probably do fine, but it's not the reality.

Will Nuxeo's programmers be happier with Java? Perhaps so, it may fit their brains well. They may find that it pays back in terms of stability, predictability and scalability (I'm somewhat sceptical of those arguments). They will certainly find it easier to hire new, talented developers.

But I can't help suspect that this has a lot to do with commercial realities. Alfresco has created a lot of noise off a product that's very, very impressive on paper and reasonably impressive on screen. It doesn't have anywhere near Plone's customisation/application-building story, for example. But if you read the spec list, it includes things like JSR170 and JSR168. Java standards (supposedly) for content storage and portlets, respectively. It talks to Lucene and Hibernate and MyFaces and MySQL, it uses Spring. These are things that are often singled out as "best-of-breed" open source components.

A digression... the Open Source in Java Enterprise world is bit of a parallel universe to the one we know in Zope and Plone. Different kinds of people hang out there. Investment comes from heavy-weights and Sun specification working groups. Apache/Jakarta are kings (and they have a lot of very, very impressive software)... end digession...

But Gartner goes out and says, "demand JSR170 to avoid lock-in". The ZODB makes IT managers quiver (call it an object store, not a database, Matt Hammilton says). Butler says, "demand JSR168 to integrate your applications with your portal server". IBM tell you, maybe it'll even work. We have viewlets and component architectures that make those things seem like last year's fashion, but that isn't what people buying IT look for. And probably for good reason. They shouldn't invest their money in unproven solutions, hard-to-integrate platforms or dying products. It's often a lot of money on the table, and being conservative is often a good thing (whether they always get what the bought is another story).

So, for Nuxeo, I'm sure that their clients will be very much more impressed if the 10 minute PowerPoint sales pitch includes a number of technologies they have heard about. It makes a lot of sense, especially since their stated target is the ECM sphere, and I'm guessing their customers have a lot of similar Java technologies already deployed. And I am really glad that there will be a credible alternative to Alfresco in the Java-based ECM/DMS market.

As far as Plone is concerned, I think our edge will always be in flexibility and agility. I can do things with Plone much faster than I can with any other platform, and I can develop cleaner, better-tested components using Zope 3 technologies than any other platform I've ever seen. At the same time, people like Enfold are doing incredible things with integration, and Plone 3.0 should put us back on the innovation curve with exciting new tools and technologies. Mostly, I'm not worried because the Plone community is so vibrant, with so many cool people and such a lot of friendship. In open source, this is worth more than technology.

But at the same time, I'm very eager to find out what Nuxeo's very talented developers will manage to come up with. Hopefully, I'll have a chance to sit down with them over a drink and get their experiences straight from the horse's mouth. Nuxeo.com/org have a commercial story to tell. I wish them the best of luck.