Why client-side XForms is not enough

About 10 years after the first version of HTML the W3C felt is was about time to create a sucessor of the HTML forms. XForms as a new open standard was born. Back then it was not surprising that XForms was emphasized to be a client-side technology which was expected to be built into the browser just like HTML forms are (this attitude changed later during the further evolution of this standard ). But XForms is not comparable to HTML forms in many ways. While HTML forms only provided some controls XForms offers a complete processing model with an MVC architecture, events, actions and a submission module and more. Without going into the details about these differences this article focusses on one aspect which is surely a prominent one for a forms framework: validation.

While some garbage data may not be a problem for a guestbook they are for a serious web application. XForms as the new standard in this area comes to the rescue. It offers a declarative way to put fine-grained constraints on data and even define dependencies between different data items. Now XForms is not natively supported in a single browser on this planet. No problem as there are several good client-side implementations around to fill that gap. So choose one of them, define your constraints and the problem is solved.

But stop – wasn’t there something?

“Never Rely on Client-Side Data Validation

Client-side validation can always be bypassed.
Server-side code should perform its own validation. What if an attacker bypasses your client, or shuts off your client-side script routines, for example, by disabling JavaScript?
Use client-side validation to help reduce the number of round trips to the server but do not rely on it for security.”

This is a cite from the OWASP (Open Web Application Security Project) site – an organisation dedicated to improving the security of applications. But maybe those guys do not know XForms already? It has a rich and fine-grained processing model that makes sure that all data stay in a consistent state and are only submitted when validation runs green. The problem here is that most of the implementations are implemented in JavaScript which can be easily bypassed if a attacker wants to and at the same time provides the toolset needed to inject malicious code into the browser.

This seems to be exactly the argument that the opposers of XForms always proclaimed: as long as XForms is not natively supported by all major browsers it will never gain importance for web application development. But that’s a fake argument. Even if browsers would support XForms natively this would probably make it harder to build an attack but you can still setup a page that sends manipulated data to the submission target. So OWASP seems to be right: ‘never rely on client-side data validation’.

How to get out of this dilemma? Is XForms altogether a bad idea? The answer is definitely no. XForms provides exactly the functionality that is needed to build reliable, flexible and future-proof applications today. It will be there for years while proprietary solutions come and go at will of their vendors. And the problem does not only apply to XForms but to any client-side technology used for data entry.

So only use server-side implementations and avoid the problem? That can be an answer and in fact those implementations have proven in many projects that server-side XForms is a real option. But server-side implementations are harder to setup and handle as the client-side ones. This raises the entry hurdles for many people. Further client-side implementations can be more responsive in some cases as they avoid roundtrips to the server that are otherwise mandatory for a full support of the XForms processing model.

How can this gap be overcome? We can go for a client-side XForms processor and replicate all validations on the server. This would of course take away much of the beauty of the declarative validation model of XForms as it forces to re-implement all validations on the server-side. And as XForms validation is so powerful it might even become a tedious, repetitive and sometimes hard to solve task. A lot of the productivity gain of XForms compared to hand-crafted script solutions would be taken away.

But the solution can be easier and more elegant. As XForms is an open standard all implementations should be interoperable meaning that the validation in one processor will lead to the same results as those from another. They are interchangeable. If we need double validation to really build secure web applications why not use a client-side processor to gather and validate the data and then send those data to a server-side processor for re-validation? All that would be needed is a reference to the original form document and the instance(s) in question. The server-side processor can then initialize the form with those instance(s) (which will include validation of the data) and forward the data to its intended URI. Or if something goes wrong send a response to the client that reports the problems found. The server-side processor would never create a UI or fire initialization events nor would it trigger any of the submission event handlers. Instead it would just forward the data or create a generic XML structure describing the validation errors.

However to make this idea a reality there are still some steps to take. An interface for the processor communication needs to be defined as well as a protocol for reporting and handling the errors. Ideally this protocol would become a part of the XForms standard itself. On the other hand XForms already has a powerful submission module that could be extended to address the described problem. E.g. a specialized URI scheme or additional property for the XForms submission element would allow to use a server-side processor to act as a proxy for a submission target.

Handling errors is a bit more complex and involves to develop a generic error-reporting structure (ideally expressed in XML) that can be interpreted by the client-side processor and results in alerts popping up or a general message explaining why submission has failed. As XForms is not most often not the only technology used to build a site or web application the using applications can also profit from such a generic error reporting by integrating it into their error page templates in cases where there is not a simple validation problem but a fatal condition that prevents a successful submission.

XForms is an open and device-independent standard and as such there will be hardly one implementation fitting all use cases and scenarios. But bridging the gaps between different implementations and let them act as one can underline the fact that open standards can accomplish much more than any proprietary solution will ever can.

Thank you for this very good analysis about limitations and security concerns for XForms client-side implementations. Automatic XML signatures could help instead of explicit hmac() function calls.

Client-side implementations are clearly lighter and can be used in various architectures such as PHP or .Net. With JSON support added, it’s even easier for todays developers.

Client processors have plenty of power waiting to be used (or wasted by Flash objects…) while networks and servers are always overloaded.

I’m still convince that XForms spirit is to be client-side implemented. Mobiles can have it natively now and double validation was already performed in old applications, by the way, because of concurrency in the database (locks and so on…).

Thank you for your comment. But please let me clarify that i’ve intentionally avoided to get into the (tiring) debate of client-side vs. server-side implementation of XForms. Clearly both approaches have their strengths and weaknesses but that’s a different discussion i really didn’t tried to open here.

What i tried to point out is that client-side XForms alone does not offer the security that is needed by serious web applications and to lay out an (admittedly rough) idea of how to improve the situation by taking together the strengths of both architectures.

I hope that both sides can work together to offer the users the best possible experience and security while providing ultimate productivity for the developers (by not unnecessarily forcing re-implementation of server-side validation).

I apologize if i should have left the impression that client-side XForms is not a fully qualified approach. But i feel that the XForms community still needs to put their brains together to convince the rest of the world.

For me, today’s XForms Specifications suggest that security has, first, to be done at HTTP protocol level with headers and encryption. For data, this can be reinforced with digest() and hmac() encoded values.

With a remote native XML Database, a standard approach is to exchange only XML data and sessions have to be modeled in the database too (I would like to have also a supported XML notation for HTTP requests and responses…).

Because of concurrent access, the database engine has, at least, to validate data against other data already in database. This can cause submission to fail for consistency reasons.

Full data validation at server-side can surely be tricky without XForms because many schema languages cannot be used for such details and interactions between values.

I understand that a server-side implementation of XForms can do its best to encrypt everything and, even, to keep sensible data at server-side. Exchanges can be done with any possible format because that hasn’t be specified.

Working with both-sides XForms implementations is a very interesting idea. It requires to improve a lot of things in the current specifications to still have not too complex XForms pages at both sides. Is it what you are thinking of?

Regarding your last comment:
yes, you got it – this is what the idea is about. I agree that the forms should not be bloated by this approach but i’m very optimistic that this part is easy. It’s probably only one additional attribute on submission (or a header or …) to specify the URI of the server-side processor (validator) in addition to the actual resource attribute that represents the final submission target.

The tricky part is to define the concrete behavior – my first guess is that actions specific to form init and submission should not fire during server-side validation. The processing should be reduced to the actual validation.

The error-handling will be the other part to think about. Error messages need to be delivered in a defined format so the client-side processor can act accordingly.

I discovered betterFORM after a circuitous Googling. I had discovered XSLTForms (client-side) and was looking for a CMS/platform or similar which is – at least mostly – based on that, because I want to setup a low-budget XForms-based site/webapp which is “non-processor-hogging” (leveraging the client-side to save hosting-costs, and because good response-time helps with SEO). I intended to gracefully degrade to a static snapshot version of the site for people with no/off javascript.

Early I realised the problem you describe about security (it *must* validate server-side), and my first thought was to implement validation (and anything else that physically can’t be done on the client, but I can’t think of yet) on the server, but offload all other XForms functionality onto the client-side implementation. To me it just seems a logical conclusion, and I was surprised that it isn’t yet a commonly used tactic. Regarding XSLTForms etc, I heard one complaint that it messes up SEO by being too javascripty for spiders, but I think a simple & fast header-sniffer-filter-layer on every page-request to the server can serve up flat-page “snapshots” to spiders, based on header-info, to avoid that. It would kill two birds with one stone, by keeping both the spiders and the no-javascripters happy (but obviously less interactive).

I was hoping to do such a set up as XRX, but I guess reconciling the integrity of that workflow with such changes would require actually hacking at both XSLTForms and betterFORM? I am still an XForms noob, so for sure I won’t even bother investigating such a task myself (yet). From reading, it seems XSLTForms is very non-intrusive to namespace, so it should play nice with betterFORM’s Dojo lib too by the way.

Am I talking nonsense? If not, I recommend this as a hobby project for some dev to try. Of course, I’m mostly selfishly motivated to see it done! (as a prospective user).

No, i mean you are not talking nonsense and i agree that this problem should be addressed. I have a solution in mind but am currently lacking time to work it out.

But you are also right when stating that this would need to hack at both sides. I hope we will find time to work on the solution sometime this year but to make sense there at least must be some interest and cooperation from some client-side implementer.

Actually for this to work we do not even need to directly integrate with the client-side implementation. Instead it would make much more sense to simply extend the wonderful XForms submission module to allow specification of a remote validator which can be called via http. By passing a reference to the original form document and the current instance data a server-side validator is able to run and validate the XForms document with exactly the same constraints as defined initially (so no code duplication or additional work to implement the second validation layer). A simple init would do to know if there are errors – if there are we send an http error code and pass validation error information (which constraint for which node failed) back. For an XForms processor this would just be a ‘validation mode’ and processing will conclude right after the xforms-ready event.

The client then will receive the http error and will handle this by dispatching a xforms-submit-error such behaving right the same to the user as if a ‘local’ validation had failed. An alert would be shown and no data will be send. The submission modules should then provide a XPath function to read the error information to give more detailed feedback. This is just one way how it could be done.

In the positive case (everything is valid) the server-side processor would then do the actual submit to the URI specified by the standard resource attribute or child. It would return a success code which results in a xforms-submit-done on the client-side.

[…] Why client-side XForms is not enough (via ) By windauer About 10 years after the first version of HTML the W3C felt is was about time to create a sucessor of the HTML forms. XForms as a new open standard was born. Back then it was not surprising that XForms was emphasized to be a client-side technology which was expected to be built into the browser just like HTML forms are (this attitude changed later during the further evolution of this standard ). But XForms is not comparable to HTML forms in many … Read More […]

I’m glad that approach is on the right track. I also have a hunch that whoever gets it to work will cause a stir, because most detractor’s comments I have read about XForms’ lack of mainstream uptake points to those two problems (client-side-only gotchas, server-side-only gotchas), both of which this solution would sweep aside simultaneously by using the best of both worlds, and only that much. And for those who wish to, using the “gracefully degrade to static pages for no-scripters and spiders” idea would sweep away the SEO complaint too. Win-win… Of course I think it would be ideal if all client-side processing could still be dealt with purely in javascript, to avoid the browser-plugin-requirement-death which XForms suffered for too long (until solutions like XSLTForms matured enough).

As a slightly parallel (almost off-topic to the article) additional thought – I have also been playing with things like “PHPRestSQL”, “Tonic” and Mysql’s “as XML” output option to see how closely I could get Mysql to emulate an xml-server (like Exist). So far, it looks surprisingly easy, but obviously considerably slower. It got me wondering – how hard would it be to create an interface to effectively do XRX on an SQL-based server?! Of course it would be *way* less optimal, but it could always be a suboptimal poor-man’s option which would for example allow someone to copy an existing betterFORM app to a server which for some reason can’t host Exist (or some other obscure limitation)… Of course it would make more sense to write/use a Java Rest-SQL layer rather than PHP, seeing that Java is already extensively used, and it would be silly to introduce a whole PHP dependency for something relatively small…

I guess these ideas are part of an approach which would allow betterFORM to be more “componentised” (is that the right word?) so that there could be the ideal combination of components, similar to what it is already, but many other combinations for those with different hardware or software setups/limitations/requirements (use of client-side XForms processing to reduce server load – or not, RDBMS/ORDBMS instead of OODBMS – or not, etc). I am aware I’m speaking as a relative noob with that stuff, so I apologise if I’ve veered into lala-land and am overlooking some obvious showstoppers about what I propose…

wow – that’s a bunch of thoughts. Though really going off-topic (btw you’re invited to start such discussions on our mailinglist) there are some interesting points i’d like to commnet on:

yes, i agree that (for the sake of the users) it would be great to see an installation makiing use of client-side and server-side processors to get the most out of it. And of course there’s no other serious option to JavaScript for the implementation. Plugins may have their use in relatively closed applicaitons but not for the broad use on the internet. To be honest i’m not sure how mature the current implementations are – at least there seem to be problems with certain features of XForms that are lacking. The most serious problem maybe being that there is no XPath 2 which will likely come up with XForms 1.2. There is an alpha version of a xquery (which includes XPath 2) implementation but i’m resist getting excited about alpha versions 😉 But when run in combination with a server-side processor that is more complete the client can rely on the back-end for the more advanced XForms features though that’s just a vision today.

Regarding the idea of a relational backend for XRX apps – yes, i think that would be possible but frankly i do find this idea very attractive for us to develop unless there pops up a customer with a real need for it. Of course there are certain requirements to run betterFORM XRX but typically it is used for serious applications and the potential server cost is not that relevant. However we also think about how to make it even easier to use betterFORM probably one day as a service in the cloud – just import one line of JS and talk to a remote XForms service for handling the XForms. But i’m going off-topic myself. Would be happy to continue these discussions on betterform-users list.

It’s the simplest thing in the world to validate a chunk of XML against an XML schema on the server side, and to build a nice, tight schema that is going to exclude rubbish.

The problem with this approach is that the validation failures returned by the xml validator engine are sometimes … a little cyrptic. XForms takes care of that. With care, you can be sure that *IF the user is using your webapp properly*, then the request XML will never fail XML validation on the server side.

That being the case, you don’t need to worry about handling parsing errors on the server side and giving helpful messages. You also don’t need to worry that the XML may cause something pathological to happen when you deserialise it ino in-memory objects. You can just put the xml validation on as the very first thing that happens to the xml before it is deserialised, throw back a 400 if it fails, and leave it at that.

Of course,there may be business-level errors, but that’s a whole different ball of wax. XML validation + XForms simlpy takes care of a whole lot of the work of helping the user to submit something basically reasonable..

i agree with you that a schema validation on the server might be a solution for the double-validation problem. However i still see some problems with that approach:
– a ready-made schema is not always around and it can take significant effort to build one. Especially for applications that start small and grow step-by-step (adding validation rules on the go) you have a ongoing effort to keep XForms validations and your schema in sync.

– XML Schema and XForms validation are not semantically equivalent. XForms is good at item-by-item validation and checking dependencies between single items (e.g. A is only valid if B has value c). These kind of constraints are important in data-entry situations but are hard to express in XML Schema (if at all).

A bit off-topic but with regard to business-level validations i do not think that we should exclude that from the discussion as otherwise you are again left with the question how to handle those. IMO XForms already has the answer to that by allowing you to call external systems (external validation services) by either using a submission or writing your own XPath extension functions that call the external service during the XForms validation process thus aligning it nicely with the item-wise XForms validations and also allowing you to display informative messages to the user and/or cancel a submission. You do not need to pass data gathered by XForms on to a ‘business-level validation layer’ for further processing but inline that into the forms directly.

This once again shows that it would be convenient to have XForms validations on the server as it would leave all originally (by the XForms author) intended constraints intact (including those expressed in XML Schema).

The main point about using XForms on both sides is to re-use the stuff you already have (the XForms document) and not forcing you to do extra work to have double validation.

I have had some success using Schematron (compiled into XSLT) for both validation and state management, though I am now moving towards using SCXML for state management with Schematron as the constraint/guard condition language to handle transitions.

With either of the approaches mentioned above the client-side and server-side validation mechanisms share the same pure XML semantics with a rather large intersection.

By intersection, I mean that typically only a subset of the validation rules applicable on the server would be shipped to the client (typically filtered via XSLT or XQuery) since, for example the client does not need to worry about simultaneous updates and record locking, nor will the client typically have the entire “model” available for complete validation. Conversely, the server will typically not be interested in UI state (in a stateless RESTfull App, i.e. no session data)

Schematron is a much closer match to XForms validation than XML Schema is. Therefore it might very well be used to drive the second validation layer. Can’t speak for SCXML as i have never used it before.

As you said you have a rather big intersection of the validation semantics with Schematron and XForms and i agree also that some validations will only happen on the server. Client-side validation should mainly avoid most of the inconveniences of old-style user interfaces and give you quick feedback while the server-side validation must ensure consistency and correctness of data.

The main point of the article was to state that you don’t get away without a server-side validation regardless of the technology used to implement this. It just would be nice for an XForms author if you could re-use the same constraints that you’ve already used on the client. This always can of course mean that the server actually validates more of these constraints than a client may probably be able to check. E.g. there are implementations that do not support Schema datatypes on the client but the server will very well deal with these. So the same form would run on the client – just checking plain XPath constraints – while the server can do the full datatype checks in addition. No extra technology or work would be needed to get a secure, double-validating architecture. I fear i have to deliver a demo one day to make the point 😉