Summary
Ruby is the cool new kid of on the block and has some significant cred in code-generation circles. Python has whitespace sensitivities that make it a poorer fit for templating solutions. This sounds like a no-brainer decision but is the real argument about which language the market wants to see used?

Advertisement

With my return to running my own business as a full-time activity and freedom to indulge entrepeneurial instincts, one of the projects I'll be putting more time into is AppMaker. AppMaker is a long-standing Mac product for application generation to which I acquired the rights when the original author, Spec Bowers, retired. Progress has been stalled for much of the time of OS/X with Apple's closed attitude on Cocoa - a contributing factor to Spec's retirement.

Given the length of time that has elapsed since there was a new release of AppMaker and the fact that very few users seemed to customise templates significantly, I feel some freedom to vary the template approach. For a start, I'm planning to replace the somewhat-idiosyncratic OO language with Python or Ruby. There's no justification for maintaining a complex language, with an old code base, when the world has moved on to create outstanding scripting and text manipulation languages. The startup cost for any new AppMaker user wanting to customise their templates will be drastically reduced if they are based on a standard language.

I was very impressed with Jack Herrington's Code Generation in Action which includes a taxonomy of code generators, and Ruby examples. His book, and the growth of Ruby in the somewhat code-generationy Rails framework, have put me in the position of having to choose - would you use Ruby or Python for a code generator language?

The AppMaker generators best fit what he calls a Partial Class Generator where a definition file (the AppMaker Project) is combined with a set of templates (the selected language/framework) to generate the code. The previous market for the product included a strong educational segment as the code generated was deliberately made representative of best practice for a given framework.

Tagged vs Mode-Flipping Templates

Many people would be familiar with JSP templates as an example of the Partial Class Generator approach. However, the AppMaker templates have somewhat of a twist and I'm trying to decide if it is a style worth preserving. Here's a reminder of the traditional bounded template that Herrington shows or JSP conditional logic:

JSP style Templates

AppMaker's template language

The AppMaker proprietary template language flips between a literal text state and a code state, using the percentage sign as the escape character to flip states. This results in templates that have less noise and probably would cause less confusion than using angle-bracketed tags like JSP. Of course, they are no longer viewable by a simple HTML browser which can ignore the JSP tags but the domains we're targetting for such generation are more likely to be generating traditional source code into multiple files such as shown (with elisions...) below.

I do not think the white space in python is a very valid argument against Python's usefulness in code generation. I used python cheetab, a template engine, a lot for code generation, and it works like a charm.

Agreed; I've used Cheetah Templates and can attest that whitespace is not a problem at all. However, I've also used PSP, where it can be quite annoying. Fortunately, with the multitude of Python web frameworks these days, I'll probably never have to use PSP again.

Err... Cheetah doesn't use Python language. I don't think anyone denies that a Python based engine can drive a templating language. But I have yet to see that the templating language is Python, in the way that JSP scriptlets contain real Java code and ERb templates contain real Ruby code. The obvious reason would seem to be that the indentation of the generated text (HTML, say) would easily interfere with the Python structure.

Whether having the template language be in the same syntax as the host is a good idea or not is of course debatable. JSP has not been astonishingly successful in a technical sense. I think ERb has a significant better result.

If there is a Python templateting language where the template syntax is Python code, I'd love to learn about it, though.

> Mr. Dent is fond of claiming that syntactically> significant whitespace is a problem but is apparently> reluctant to back it up.

S'funny - two posts in a row and I'm now fond of claiming. I'm actually a fan of Python as a programming language, especially because of how well the whitespace works.

However, I am very much aware of the problems with whitespace in mixed Python programming environments (painful experience) and with copying and pasting code from email and web pages. I can't back this up with examples because they are all transitory experiences within a programming team I no longer belong to and wouldn't have the right to disclose anyway.

When it comes to markup languages, as reference my recent doubts over YAML, I stand by my experience. Significant whitespace in casually edited files with insufficient redundancy is a solution that doesn't scale. If you want to look into some of the complex XML-based interoperability projects from which I derive these opinions, have a waltz through https://www.seegrid.csiro.au/.

Johannes wrote> ... I have yet to see that the> templating language is Python, in the way that JSP> scriptlets contain real Java code and ERb templates> contain real Ruby code. The obvious reason would seem to> be that the indentation of the generated text (HTML, say)> would easily interfere with the Python structure.

Absolutely. You have understood and explained the issue, clarifying what I didn't. Thanks.

> If there is a Python templateting language where the> template syntax is Python code, I'd love to learn about> it, though.

> > Mr. Dent is fond of claiming that syntactically> > significant whitespace is a problem but is apparently> > reluctant to back it up.> > S'funny - two posts in a row and I'm now fond of> claiming. I'm actually a fan of Python as a> programming language, especially because of how well the> whitespace works.

I find it a little annoying to directly question someone about an assertion, receive no response and then see a similar assertion posted by that same person days later. But, I do regret the tone of my post above all the same.

> However, I am very much aware of the problems with> whitespace in mixed Python programming environments> (painful experience) and with copying and pasting code> from email and web pages. I can't back this up with> examples because they are all transitory experiences> within a programming team I no longer belong to and> wouldn't have the right to disclose anyway.

I don't need proof. I just think you could explain exactly how things go wrong and what issues were caused. Then we could discuss whether other constraints would resolve those issues. Until I know for sure what the issues are, however, it's difficult to come up with solutions.

I can understand the issue with email but I don't find that to be a very compelling reason.

> When it comes to markup languages, as reference my recent> doubts over YAML, I stand by my experience. Significant> whitespace in casually edited files with insufficient> redundancy is a solution that doesn't scale.

Honestly, I don't really have a strong prejudice for significant whitespace. What I really care about is required consistent indentation. But it's frustrating to keep hearing about how significant whitespace is an issue without a clear explanation. My personal experience is that it's never been an issue. There are a lot of ways to destroy source. That's why source control exists. I'm not swayed that significant whitespace is so fragile that it should not be used as a technique.

> If you want> to look into some of the complex XML-based> interoperability projects from which I derive these> opinions, have a waltz through> https://www.seegrid.csiro.au/.

I'm not sure I understand how XML is relevant to the issue at hand. Perhaps you have a more direct link.

For code generation I used to prefer Cheetah but now I use Mako [1]. It's syntax is simpler and it's coding syntax matches Python except for the '%' marker and the end of code markers. It's just as fast and capable as Cheetah and it's documentation is very clear and covers all features.

I wonder why all those template languages look like language design accidents or first year student experiments? Maybe the ugliness is intended and instead of gluing two languages together so accurately that no one notices the glue the glue becomes part of the visual design and the look shall stick to all the funny % characters because they spot the interesting things.

Moreover '{%' or '<%' like parens are idiomatic for template languages because no tasteful language designer would ever use them in their language design: ugliness as a simple disambiguation strategy.

> I wonder why all those template languages look like> language design accidents or first year student> experiments? Maybe the ugliness is intended and instead of> gluing two languages together so accurately that no one> notices the glue the glue becomes part of the visual> design and the look shall stick to all the funny %> characters because they spot the interesting things.> > Moreover '{%' or '<%' like parens are idiomatic for> template languages because no tasteful language designer> would ever use them in their language design: ugliness as> a simple disambiguation strategy.

while I'm personally partial to the % based approach (I wrote Mako), I think you're looking for http://genshi.edgewall.org , another great approach.

in short i'm a little surprised that the original post here did not mention any of the currently popular Python templating languages...if PSP, which is not widely used at all, is all that could be found, I fear we aren't advocating for Python strongly enough.

James Watson wrote:> > However, I am very much aware of the problems with> > whitespace in mixed Python programming environments> > (painful experience) and with copying and pasting code> > from email and web pages.> > I don't need proof. I just think you could explain> exactly how things go wrong and what issues were caused.

Sorry, I thought the Python issues were sufficiently well-known I didn't need to elaborate. The problems generally come from mixing tabs and spaces, tools which react differently to the tab key and people moving between tools and sharing code across environments.

A common recommendation for Python programmers is to disallow tab characters in source and only allow for an agreed number of spaces per indent. The typical ways in which an indent is inserted or adjusted in front of code are:

- manual insertion of X number of spaces- pressing the tab key to insert an "indent" (often translated as X spaces)- pressing shift-tab to remove one indent level- pressing other key combinations to control indent/outdent- hitting return at the end of a line and having the auto-indent position your next line

Most source editors allow you to set the tab key or their indent/outdent facility to thus use that number of spaces. I've encountered some situations, particularly where keys other than tab are used for indent/outdent, where the tab key will still insert tabs. If I'm correct in recalling this, one particularly subtle behaviour I've seen is that the tab key inserts spaces (adjusting indent) when it is at the start of the line or if the entire line or lines are selected but will still insert plain tabs when the insertion point is within a series of spaces.

Whatever the cause, I've definitely seen umpteen examples of source code that have ended up with mixed spaces and tabs doing the indent. Python copes to some degree but can be unpredictable and often complain about such source.

Depending on the email clients used, quoting is sometimes shown as indentation and tabs are translated into varying numbers of spaces. I'm talking about people on Mac, Windows and Linux clients here.

IBM's DeveloperWorks, if I recall correctly, has often published web articles where the Python source example fragments have been mangled by their whitespace processing and wrapped unmercifully. Usually the source is available as an attached zip file but copying and pasting quick examples in such circumstances requires reading the source and working out the apparent logic and thus where indents lie.

In a language like Python with syntactical clues and being able to interpret the logic of the source, an experienced programmer can usually deduce the correct indent levels and logic nesting. The start of class and function definitions are unambiguous so the main point of concern is the local nesting into loops and conditionals.

> But it's frustrating to> keep hearing about how significant whitespace is an issue> without a clear explanation. My personal experience is> that it's never been an issue. There are a lot of ways to> destroy source. That's why source control exists.

I have seen confusion over diffs between versions because the diff tool had been configured to ignore leading whitespace, in the interests of reducing noise levels when comparing C++ code, in ignorance of the effects on Python source comparison.

I'm not talking so much about destroying source as getting it wrong to start with when transferring from other sources into your program. Hopefully the above scenarios clarify my point.

> I'm not swayed that significant whitespace is so fragile that> it should not be used as a technique.As I said in my previous post, when it comes to Python code I'm a fan of the language and I think the huge benefits of the significant whitespace override the occasional hiccups in a team. If you have a mixed environment it is important to be wary of the whitespace issues. PyLint has default configurations to pick up mixed spaces and tabs.

When it comes to storing structured data, the reason I recommend XML (or any other format with named closing structures) is that complex data often lacks the redundancy in content that clues us to indentation levels being incorrect and thus nesting being wrong. If you have nested levels of items with arbitrary property lists, how do you tell if a given key-value pair belongs to a parent structure other than solely by the indent level?

Kay Schluehr wrote:> Moreover '{%' or '<%' like parens are idiomatic for> template languages because no tasteful language designer> would ever use them in their language design: ugliness as> a simple disambiguation strategy.

Giggling thanks, Kay.

I think the vast majority of template languages are invented as an attempt to redo JSP or end up looking like that through the same thinking process - make the tags look like elements so people used to hand-editing HTML will understand them.

Whilst I appreciate its power, I've never enjoyed programming in XSLT and shudder at some of the other attempts to write programming languages in XML.

One of the distinguishing characteristics I think of using a template language like AppMaker's, to generate programs, is the interweaving of comparatively large amounts of template source with the literal text. This is very easy to edit with its simple flipping modes compared to the tags for bounding embedded stuff approach of most of the web-oriented template languages.

The desire to replace the homegrown OO language in the templates comes from a maintenance need and a marketing suspicion that a more recognisable scripting language would attract more people to modify templates.

Mike Bayer wrote> i'm a little surprised that the original post> here did not mention any of the currently popular Python> templating languages.

I mentioned PSP because I had a link to it in my notes from research about three years ago and it was representative of the kind of templating syntax out there. I didn't mean to imply any dominance or endorsement. Thanks for the pointer to Genshi and I particularly appreciated its plain text mode:

Dear $name,

These are some of my favorite fruits:#for fruit in fruits * $fruit#end