Even if this can't be fixed in the parser, the documentation should be updated to define a preferred idiom for obtaining a string containing a single backslash.

jon.

note: this issue partially duplicates a comment by Guido Deinhammer on issue #454 regarding other escaping issues. I have raised a separate issue, so that this one can be addressed separately, if desired.

Even though it's possible to use the single-quotes to get a single backslash, escaping within double-quotes does seem a bit inconsistent.

Given that the backslash-escaping idiom has been adopted from programming languages, it would be nice to be consistent with their behavior. That is, ought to yield \ in all cases, possibly including single-quoted strings.

Christopher Schultz
added a comment - 11/Feb/08 17:21 Even though it's possible to use the single-quotes to get a single backslash, escaping within double-quotes does seem a bit inconsistent.
Given that the backslash-escaping idiom has been adopted from programming languages, it would be nice to be consistent with their behavior. That is, ought to yield \ in all cases, possibly including single-quoted strings.

There's plenty of inconsistencies and shortcomings in Velocity's escaping support. Personally i never use or recommend the use of \ for escaping in either general template content nor interpolated strings (where it is most broken!) for that reason. Of course, this years-long practice has significantly reduced my motivation to help remedy this, as it is reflexive to avoid the matter entirely.

On second thought, there is a bug here, in that neither #set( $foo = "\" ) nor #set( $foo = "\." ) should cause the lexical errors that they do, though that is somewhat different than title. I suppose i should read more thoroughly. Of course, this is Monday and i've loads of email to rush through... :-/ Sorry, Jon. You've got a bug here. I'll just change the title to fit better...

Nathan Bubna
added a comment - 11/Feb/08 18:16 There's plenty of inconsistencies and shortcomings in Velocity's escaping support. Personally i never use or recommend the use of \ for escaping in either general template content nor interpolated strings (where it is most broken!) for that reason. Of course, this years-long practice has significantly reduced my motivation to help remedy this, as it is reflexive to avoid the matter entirely.
On second thought, there is a bug here, in that neither #set( $foo = "\" ) nor #set( $foo = "\." ) should cause the lexical errors that they do, though that is somewhat different than title. I suppose i should read more thoroughly. Of course, this is Monday and i've loads of email to rush through... :-/ Sorry, Jon. You've got a bug here. I'll just change the title to fit better...

Etienne, thanks for pointing out the use of single quotes - I should have read the doco better. If I had had my shell-script hat on this would have occurred to me but since I didn't have the VTL reference handy when I was struggling with this issue, I had incorrectly assumed that '\' would be treated as a Java character reference.

Nathan: I guess it doesn't really surprise me that #set($foo="\") is a parsing error, if \'s are meant to be treated as escape characters when parsing strings. More surprising was the failure of #set($foo="\.") since based on other experiments I expected the \ to be preserved.

It would be good to have some clarity on what \ is meant to do in double quoted strings since I don't think (but I may be wrong) this is specified clearly anywhere.

Perhaps it would be useful to include a section about the behaviour of backslashes in double quoted strings in the user's guide?

Jon Seymour
added a comment - 11/Feb/08 21:51 Etienne, thanks for pointing out the use of single quotes - I should have read the doco better. If I had had my shell-script hat on this would have occurred to me but since I didn't have the VTL reference handy when I was struggling with this issue, I had incorrectly assumed that '\' would be treated as a Java character reference.
Nathan: I guess it doesn't really surprise me that #set($foo="\") is a parsing error, if \'s are meant to be treated as escape characters when parsing strings. More surprising was the failure of #set($foo="\.") since based on other experiments I expected the \ to be preserved.
It would be good to have some clarity on what \ is meant to do in double quoted strings since I don't think (but I may be wrong) this is specified clearly anywhere.
Perhaps it would be useful to include a section about the behaviour of backslashes in double quoted strings in the user's guide?
jon.

If you ask me, the only thing in double-quoted strings that should behave differently than it does in a template in general is the double quote (which ends that string and therefor cannot currently be directly put in double-quoted string). A backslash in a double string should follow the same rules as a backslash in a template in general (i.e. only meaningful for escaping valid references/directives/macros). It definitely shouldn't be a parsing error and ideally should not warrant a section in the user's guide.

Nathan Bubna
added a comment - 11/Feb/08 22:09 If you ask me, the only thing in double-quoted strings that should behave differently than it does in a template in general is the double quote (which ends that string and therefor cannot currently be directly put in double-quoted string). A backslash in a double string should follow the same rules as a backslash in a template in general (i.e. only meaningful for escaping valid references/directives/macros). It definitely shouldn't be a parsing error and ideally should not warrant a section in the user's guide.

IMO, you should be able to be a double-quote into a doubly-quoted string. It sounds like Nathan is saying that the backslash should only be used to escape references and directives/macros. I think the backslash should be extended to act like a more general escape character. For instance, I would expect this to work:

#set($doubleQuote = "\"")

Instead of $doubleQuote containing a single double-quote, it contains a backslash plus a single double-quote (at least in Velocity 1.4).

I would think that it would actually be easier to make the parser treat backslash as a general escape character than to have different rules in different places (the different rules being that backslashes are consumed in some circumstances but not in others).

Christopher Schultz
added a comment - 12/Feb/08 14:21 IMO, you should be able to be a double-quote into a doubly-quoted string. It sounds like Nathan is saying that the backslash should only be used to escape references and directives/macros. I think the backslash should be extended to act like a more general escape character. For instance, I would expect this to work:
#set($doubleQuote = "\"")
Instead of $doubleQuote containing a single double-quote, it contains a backslash plus a single double-quote (at least in Velocity 1.4).
I would think that it would actually be easier to make the parser treat backslash as a general escape character than to have different rules in different places (the different rules being that backslashes are consumed in some circumstances but not in others).

We've had a lot of debates on escaping issues over the years. Feel free to search the archives if you'd like to be caught up on the various arguments. But perhaps it will suffice to point out that the only escaping solutions for strings (double or single quoted) that we have ever achieved a consensus on are VELOCITY-520 and VELOCITY-555. 520 was quickly patched, but sadly, Geir never shared the patch for 555 and no one has stepped up to implement it themselves yet.

Things like VELOCITY-519 may be considered if/when someone starts work on Velocity Engine 2.0, but not before. And when that time comes, you can be sure that i will always be loudly advocating for the maximum possible consistency between parsing done in interpolated strings and parsing done elsewhere. If we do make the backslash a general escape character, you can bet i'll argue to have it work the same everywhere.

Nathan Bubna
added a comment - 12/Feb/08 16:56 We've had a lot of debates on escaping issues over the years. Feel free to search the archives if you'd like to be caught up on the various arguments. But perhaps it will suffice to point out that the only escaping solutions for strings (double or single quoted) that we have ever achieved a consensus on are VELOCITY-520 and VELOCITY-555 . 520 was quickly patched, but sadly, Geir never shared the patch for 555 and no one has stepped up to implement it themselves yet.
Things like VELOCITY-519 may be considered if/when someone starts work on Velocity Engine 2.0, but not before. And when that time comes, you can be sure that i will always be loudly advocating for the maximum possible consistency between parsing done in interpolated strings and parsing done elsewhere. If we do make the backslash a general escape character, you can bet i'll argue to have it work the same everywhere.